Implementing and Evaluating Interventions to Improve School Readiness and Early Literacy Experience from the Pacific Early Age Readiness and Learning Program Operational Guide Binh Thanh Vu, Sandra Beemer, Sally Brinkman, Wendy Jarvie, Kevin Macdonald, Myrna Machuca-Sierra, Kris McDonall, Alanna Sincovich Report No: AUS0000978 . Pacific Islands Pacific Early Age Readiness and Learning Implementing and Evaluating Interventions to Improve School Readiness and Early Literacy Experience from the Pacific Early Age Readiness and Learning Program . June 2019 . EDU Cover Image: Quang Vinh Nguyen Supplementary Annexes Cover Image: Conor Ashleigh Document of the World Bank . . © 2019 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2019. Implementing and Evaluating Interventions to Improve School Readiness and Early Literacy. © World Bank.” All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Implementing and Evaluating Interventions to Improve School Readiness and Early Literacy Experience from the Pacific Early Age Readiness and Learning Program Operational Guide Binh Thanh Vu, Sandra Beemer, Sally Brinkman, Wendy Jarvie, Kevin Macdonald, Myrna Machuca-Sierra, Kris McDonall, Alanna Sincovich Acknowledgements This document brings together five years of knowledge generated under the Pacific Early Age Readiness and Learning (PEARL) Program, which was funded through the Global Partnership for Education (GPE). The program would not have been possible without the support and generosity of GPE. The authors would like to thank the World Bank management for their support and guidance to ensure the lessons from PEARL could be documented and disseminated through this report. We would also like to thank the peer reviewers, Luis Crouch (Senior Economist, Research Triangle Institute) and Sangeeta Dey (Senior Education Specialist, World Bank) for their valuable comments and inputs to this document. We would especially like to acknowledge the support of the Tonga Ministry of Education and Training, the Kiribati Ministry of Education, the Tuvalu Ministry of Education, Youth and Sports, the Samoa Ministry of Education, Sports and Culture, and the Solomon Islands Ministry of Education, Human Resources and Development, who made it possible to carry out the PEARL activities described in this report. The interventions in Kiribati, Tonga and Tuvalu could not have happened without the ongoing commitment and dedication of the PEARL teams assigned within each ministry. There were many other stakeholders in each of these countries, as well as other Pacific countries, who are too many to list. Their commitment to improving the school readiness and early literacy of their children is why this program was successful and this document is possible. The PEARL program also benefited from the expertise of individuals and firms around the globe responsible for carrying out surveys, and designing, implementing, monitoring and evaluating the pilot interventions. There were many members of the World Bank PEARL team who contributed countless hours to the program, including experts in playgroups and early reading. Their high-level of commitment and inputs have been invaluable in ensuring the program’s success. Finally, a very big thank you to all of the children, teachers, school leaders, families and communities who participated in surveys and interventions. The PEARL program and this document has learned and benefited immensely from your involvement. ii Abbreviations and Acronyms CAPI Computer Assisted Personal Interview CLRW Come Let’s Read and Write COS Classroom Observation Snapshot COS-C Classroom Observation Snapshot for Coaching COS-M Classroom Observation Snapshot for Monitoring CPBA Community Play-Based Activity DJ Disc Jockey ECD Early Childhood Development ECDE Early Childhood Development and Education ECE Early Childhood Education EDI Early Development Instrument EFA-FTI Education for All – Fast Track Initiative EGR Early Grade Reading EGRA Early Grade Reading Assessment eHCI Early Human Capability Assessment L1 Language One L2 Language Two MET Ministry of Education and Training MEYS Ministry of Education, Youth and Sports OECD Organisation for Economic Co-operation and Development PASEC Programme d’analyse des systèmes éducatifs de la CONFEMEN PEARL Pacific Early Age Readiness and Learning PIC Pacific Island Country PILNA Pacific Islands Literacy and Numeracy Assessment PISA Programme for International Student Assessment PNG Papua New Guinea PPVT Peabody Picture Vocabulary Test PRAT Pupil Rapid Assessment Test SABER Systems Approach for Better Education Results SRS School Readiness Support (Officers) TeHCI Tonga Early Human Capability Index TG Teacher Guide Tu-eHCI Tuvalu Early Human Capability Index iii Table of Contents Acknowledgements....................................................................................................................................... ii Abbreviations and Acronyms ....................................................................................................................... iii Executive Summary....................................................................................................................................... 1 A. Overview ........................................................................................................................................... 1 B. School Readiness Intervention: Playgroups ...................................................................................... 3 Resources .............................................................................................................................................. 4 Venues ................................................................................................................................................... 5 Facilitators............................................................................................................................................. 5 Mentoring/coaching facilitators ........................................................................................................... 6 Monitoring and Supporting Playgroups ................................................................................................ 6 C. Early Grade Reading Intervention ..................................................................................................... 8 EGR instructional and assessment resources ........................................................................................ 9 Staffing Arrangements for Resource Development ............................................................................ 10 Teacher Training and Coaching........................................................................................................... 10 Monitoring and supervision of EGR program activities ...................................................................... 11 D. Alignment of interventions ............................................................................................................. 13 Chapter 1 – Social and economic arguments for investing in school readiness and early literacy ............ 14 A. Good education leads to good outcomes for individuals and societies ......................................... 14 B. Investment in school readiness and early literacy improves learning outcomes for children ....... 16 C. Public investment is essential ......................................................................................................... 16 D. Children without solid foundations start behind, and stay behind ................................................ 16 E. Specific teaching practices and play-based learning provide a solid foundation ........................... 17 F. Conclusion ....................................................................................................................................... 19 Chapter 2 – Designing Interventions to Promote School Readiness .......................................................... 20 A. How to identify interventions to improve school readiness........................................................... 20 What works to promote early child development and school readiness outcomes?.......................... 20 Assessing governance, institutional and policy capacity to promote school readiness ...................... 23 Assessing community capacity to promote school readiness ............................................................. 24 Measurement of Early Childhood Development (ECD) outcomes ....................................................... 24 Key considerations for choosing school readiness interventions ........................................................ 26 Deciding on the scale, phasing and number of interventions ............................................................. 28 B. Designing and implementing a multi-level intervention for school readiness ............................... 29 Child/Family-level intervention: Playgroups ....................................................................................... 29 Community-level interventions: National Awareness Campaigns ...................................................... 36 iv System/Policy-level interventions: Supporting partnerships in Early Childhood Education ................ 40 C. The value in measuring school readiness and disseminating results ............................................. 44 Choosing the right instrument ............................................................................................................ 45 Census versus sample? ........................................................................................................................ 49 Government counterparts and ministries ........................................................................................... 49 Data collection .................................................................................................................................... 50 Data analysis ....................................................................................................................................... 50 The importance of disseminating results ............................................................................................ 51 D. Conclusions ..................................................................................................................................... 54 Chapter 3 – Designing Interventions to Improve Early Literacy ................................................................. 55 A. How to identify interventions to improve early literacy? ............................................................... 55 What works to improve early literacy skills? ...................................................................................... 55 Key elements of early literacy ............................................................................................................. 55 Act on evidence - Assessing early grade reading skills........................................................................ 57 Setting objectives and priorities for early grade reading program ..................................................... 58 Deciding on the scope and scale of an intervention ........................................................................... 59 B. Designing and implementing multi-level interventions for early grade reading ............................ 60 Learning to read is influenced by factors at many levels .................................................................... 60 Components of a multi-level early grade reading intervention .......................................................... 62 Instructional approach ........................................................................................................................ 63 Teacher training and continuous support ........................................................................................... 68 Monitoring and supervision ................................................................................................................ 72 Community engagement and awareness (community level).............................................................. 75 C. Monitoring and Measuring EGR programs ..................................................................................... 76 Monitoring the program implementation .......................................................................................... 76 Measuring the program outcomes ..................................................................................................... 77 Bringing results to policy ..................................................................................................................... 79 D. Conclusions ..................................................................................................................................... 79 Chapter 4 – Key considerations for scaling up ............................................................................................ 81 A. Defining, monitoring and evaluating program objectives .............................................................. 81 Program objectives ............................................................................................................................. 81 Measuring outcomes .......................................................................................................................... 81 Measuring and monitoring implementation....................................................................................... 82 Evaluation and review strategy .......................................................................................................... 83 v B. Human resource planning ............................................................................................................... 83 Staff roles and responsibilities ............................................................................................................ 83 Assessing and staff capacity ............................................................................................................... 84 Training ............................................................................................................................................... 84 C. Costing ............................................................................................................................................ 85 Staffing requirements and time .......................................................................................................... 85 Transportation costs ........................................................................................................................... 85 Material costs ..................................................................................................................................... 86 Public campaign .................................................................................................................................. 86 Financial sustainability ........................................................................................................................ 86 D. Legislative, policy and regulatory requirements............................................................................. 87 Public servants .................................................................................................................................... 87 Government budget ............................................................................................................................ 87 Responsibilities of ministries ............................................................................................................... 87 Curriculum ........................................................................................................................................... 87 Regulatory framework ........................................................................................................................ 88 Flexibility ............................................................................................................................................. 88 E. Planning and scale-up steps for the PEARL interventions .............................................................. 88 References .................................................................................................................................................. 91 Annexes ....................................................................................................................................................... 99 vi Executive Summary A. Overview There is now broad consensus that progress in reducing poverty and fostering shared prosperity can be considerably enhanced through human development. In recent decades, Pacific Island Countries (PICs) have improved school enrollment substantially. As a result of PIC governments’ concerted efforts, and in partnership with donors, non-government organizations and other development partners, many countries have increased and sustained net primary enrollment rates well above 90 percent as well as large increases in secondary schooling since the 1990s. In most countries, girls’ enrollment rates are virtually the same as those of boys; even in those countries where girls’ enrollment rates are trailing, there has been a continuous upward trend. However, like many other regions in the world, the PICs face a “learning crisis”: despite high enrollment rates, learning outcomes are low. Data on student achievement among PICs show many children are completing primary education without basic literacy and numeracy skills. Weak foundations of learning are also behind weak performance mid-way into primary. The 2015 Pacific Islands Literacy and Numeracy Assessment (PILNA) identified the proportion of students performing at or above the regional benchmark level of proficiency for reading was only 46 percent for both grade 4 and grade 6 students. While children are performing better on numeracy than literacy in primary school, PILNA results show children are falling behind as they progress. The percentage of children performing at or above the expected proficiency level in mathematics in 2015 was 86 percent in Year 4, dropping to 68 percent by Year 6. While children are learning some mathematics as they progress from Year 4 to 6, these results show most children are learning the basics of numeracy by the time they reach the middle grades of primary but are starting to fall behind expectations as more difficult concepts are introduced before they reach the end of primary school. These results raise questions not only about the formal education systems, but about the preparedness of children to succeed in school. PICs’ spending on education is high for countries with their levels of Gross Domestic Product per capita, and, in the Cook Islands, Federated States of Micronesia, Kiribati and Solomon Islands, it is exceptionally high even compared to much richer countries. However, this high overall spending is not consistently converted into high spending on early childhood development (ECD). Moreover, provision of high-quality early childhood education (ECE) services in the Pacific region faces considerable challenges. From 2009 to 2012, the World Bank, with funding from the Education for All – Fast Track Initiative (EFA-FTI)1 and the Australia Department of Foreign Affairs and Trade, conducted a number of analytical pieces in the Pacific to better understand the factors impacting on poor learning outcomes of students. These included: early grade reading assessments (EGRAs) in Tonga, Vanuatu and Papua New Guinea to identify weaknesses in student reading and possible contributing factors; assessments of the policy and enabling environment for early childhood development activities using the Systems Approach for Better Education Results (SABER) tools; and community dialogs to better understand perceptions around school 1 EFA-FTI is now known as the Global Partnership for Education 1 readiness and early learning. Some of the key results highlighted: • Many parents and caregivers placed low value on early childhood development interventions, and there were not adequate systems in place in most Pacific countries to support programs. • After three years of schooling, only 3 out of 10 Grade 3 students in Tonga and 1 out of 4 Grade 3 students in Vanuatu (both Anglophone and Francophone streams) were able to read fluently for comprehension. • The low efficiency of education systems could be partly explained by poor alignment between curriculum, pedagogy and resources supporting the teaching and learning of reading. • Children come to primary education with varying degrees of school readiness, and individual and household factors are important drivers of variation in reading ability at school. Consequently, school readiness seemed to be a crucial constraint to child literacy in these countries. In 2014, the Global Partnership for Education provided funding to the World Bank to design and implement the Pacific Early Age Readiness and Learning (PEARL) program to improve Pacific countries’ capacity to use evidence to improve school readiness and early grade literacy. The measurement of school readiness outcomes and early grade literacy skills as well as the design of interventions to improve these were supported in Kiribati, Samoa, Solomon Islands, Tonga and Tuvalu. In Tonga, the implementation and evaluation of interventions were also supported using a rigorous randomized control trial impact evaluation. The implementation of the PEARL program has generated significant data, evidence, operational experience and knowledge. One of the key challenges in developing its surveys and interventions was a lack of guidance or documentation on how to do so. Many of the PEARL activities were unique and innovative, but the process of their design and implementation is highly applicable to many other contexts. They drew on global experiences, but were adapted to local contexts, using carefully designed monitoring and evaluation systems to understand what was happening on the ground and amend accordingly. This document aims to draw on the experiences of design and implementation of PEARL activities in order to provide guidance to practitioners who plan to adopt or implement similar activities in other countries, particularly in the Pacific and similar contexts. In addition to implementation details, the document includes information about the outcomes of the activities, justification for choosing the activities, and lessons learned. The document is divided into four chapters, with an executive summary that provides the main points of the “how to” design, implement and monitor pilot interventions of school readiness and early literacy. Chapter 1 provides arguments, particularly to funding agencies, on the benefits of investing in school readiness and early literacy. Chapters 2 and 3 provide more detail on the theory and evidence base for the selection of school readiness and early literacy interventions under PEARL. Chapter 4 provides information on key elements to be taken into consideration when moving from pilot to scale with the aforementioned interventions. Before designing the pilot interventions, a crucial step is to identify the status of child 2 development outcomes and/or reading level. Equally important is determining how this will be done, and the measurement tool to use. The purpose and goals behind the measurement endeavor is key in helping to choose the type of tool that is needed. For example, a tool used to assess the impact of an early years’ intervention will need to measure the skills that are being targeted through the intervention (e.g. literacy, numeracy, or school readiness more broadly) and will require the sensitivity to pick up changes in these skills over time. A tool that is most appropriate for use at a population level on the other hand, will need to be feasible for use on a large scale (e.g. quick and simple to administer, low in cost). This is described in detail in Chapters 2 and 3 under the respective sections on measurement, but highlighted here to explain that each of the interventions described below were informed by the findings from such assessments in each country and adapted to the local custom, cultural and capacity context. B. School Readiness Intervention: Playgroups The process for selecting school readiness interventions appropriate for the local context draws on multiple sources of information obtained locally, regionally and globally. It is important to apply rigorously-proven successful interventions in similar contexts, but more importantly to adopt those based on local data and evidence. Tonga and Tuvalu presented two different cases of what was happening in each country in the space of early childhood development prior to the introduction of PEARL interventions, most notably the 95 percent (in 2016) coverage of preschools in Tuvalu compared to an estimated 25 percent (in 2014) in Tonga. Data collected under PEARL in each country led to the decision to pilot playgroups in each country, but using different modalities. In Tonga, communities were supported to set up and run playgroups with caregivers and children, while in Tuvalu, preschool teachers were trained and resourced to carry out playgroups. Community-run playgroups were selected in Tonga and Tuvalu for several reasons as described in greater detail in Chapter 2. A series of community dialogues conducted in Tonga, Tuvalu and other Pacific Island Countries identified the need for strengthening parents’ awareness of the value of school readiness activities as well as the significant potential for communities, including local leaders and parents’ peers, to influence parenting. School readiness assessment surveys (EHCI) conducted subsequently also found that quality caregiver-child interactions were infrequent for many young children. Community-run playgroups offered a modality to mobilize the potential of communities to improve parenting practices. Playgroups had recently been shown to be effective in Australia and Indonesia, and earlier evidence from community mothers’ groups to promote feeding practices in Africa demonstrated the effectiveness of community, peer learning approaches more generally. Alternative modalities, including home visits, were considered to be less able to mobilize communities’ substantial potential to influence parenting. In addition, community-run playgroups were designed to be very low cost, relying on communities for resourcing apart from training and mentoring of volunteer facilitators, to enable reaching as many children as possible. This was especially important in Tonga where preschool enrollment was quite low, at around 25 percent. Playgroups consist of regular (minimum weekly) gatherings of caregivers with their children under school age (typically 10-15 families) coming together to interact and engage in play-based learning activities that provide a stimulating environment for children, an opportunity for caregivers to interact with their children in this manner, and a learning opportunity for caregivers to continue engaging with their children daily. The gatherings, typically lasting two hours, are facilitated by individuals trained to run 3 the sessions. These could be parents, community members, teachers, or anyone else with an interest in early childhood development. They are run in a cooperative spirit, with parents involved in decisions on frequency of sessions, rules, and types of activities. They are built on the proposition that early childhood programs are most effective when they support parents’ and other caregivers’ active participation in their children’s early learning and development. So, caregivers are also provided with ideas about activities they can do at home with their children, thus increasing the exposure of the children to development activities, while also increasing bonding time between caregivers and children. Playgroups are a low-cost intervention, and around the world they are known as being suited to the needs of parents and younger children and can be tailored to meet the needs of different communities. Costs of establishing and operating playgroups can be kept low by recruiting volunteers to facilitate sessions, using an existing venue to operate in rather than a purpose-built facility, and using local materials for play and learning activities. Playgroups harness community strength and build on existing support for ECD. In Tonga, each community playgroup - known locally as a community play-based activity (CPBA) - was facilitated by a volunteer community member and supported by a village early childhood education committee or a local church. Tongan communities and community-based organizations including churches have long been the financers of preschools, with financing provided by tuition from parents, donations from community members (including those who had emigrated), community volunteers, and donations from church groups. Building on this existing support for ECD was an important factor in being able to establish and operate CPBAs in Tonga. Resources Availability of funding or lack thereof can have large implications for the kinds of activities that are engaged in during playgroup sessions, though one of the benefits of playgroups is the wide variety of interesting and engaging activities that can be carried out with a very small budget. As much as possible, local resources can be gathered, such as sand and other items found in nature, old unwanted household materials, etc., which not only reduces costs, but also ensures children are able to easily identify with the materials and activities they are engaging in. It may require some costs in the beginning to get started (though these do not need to be expensive either), but the main focus of identifying resource requirements is on play-based interactions with young children, allowing them to interact with other children, adults and the environment around them in a way that stimulates their senses. It’s also important they are suitable for young children, such as not containing choking hazards or toxic substances. Under PEARL, the program developed a ‘starter pack’ for all playgroup facilitators, which included a variety of materials sourced internationally and locally. Activities were designed to make use of items that can be found in the local supermarket. The starter kit included: beading and lacing kits, building blocks, toy cars, bubble sets, playdough, children’s books in local language, a variety of stationery items, a set of recipe cards to make new materials and activities (e.g. for bubbles, goop, playdough, finger paint), groceries procured locally to make the recipes, and a set of fact sheets (Annex 1) for facilitators on how to run CPBAs. In addition to this starter kit, many communities provided additional materials, including tables, chairs, mats, cupboards, playground equipment, and additional materials for the play-based activities. 4 In many PICs, one of the biggest constraints to learning to read in mother tongue is lack of availability of reading materials in local languages. This is the case in Tonga, and PEARL sought to provide and produce more reading materials in Tongan language for caregivers to read with their young children. Ministry of Education and Training (MET) staff developed Tongan language reading books that were printed under PEARL, and additional books in Tongan were sourced from the University of the South Pacific in Fiji. Venues Identifying the requirements for suitable venues is important, as there are likely to be different options available across communities, but not all will be suitable for such activities. It’s key to understand what the requirements of the venue are, such as a safe, weather proof structure with sanitation facilities, and preferably a suitable outdoor space for children to play. The venue should be available for use by the playgroup as per a regular agreed schedule, without being asked to regularly change date or time, or move elsewhere. If such requests occur often, it can be difficult to retain interest from families. The responsibility for providing a suitable venue for the playgroups to run will be a collaboration between stakeholders, such as government, churches and communities. Having identified the minimum requirements for a suitable venue during the design phase, these will be provided to relevant stakeholders to help select a venue in each community. In Tonga, venues included community halls, church halls, primary schools and private residences, while in Tuvalu they were held in existing preschools. Facilitators The process for selecting facilitators will require consensus from the stakeholders in consultation with local communities. Funds may exist for a stipend or salary, or facilitators may be unpaid volunteers. Irrespective of these conditions defined by the community, broadly speaking, facilitators should be committed to and passionate about working with families and particularly children, dedicated enough to commit to a regular schedule, be open to new ideas, and be prepared to undertake training in facilitating the groups. Teachers (retired, part-time or preschool), health workers (such as community nurses), and social or community workers are often good choices for facilitators, as they have the experience and desire to work with families and help them towards better caregiving practices and child development outcomes. In Tuvalu, the facilitator role became an extension of the preschool teacher’s role, opening their preschool to caregivers and younger children at least one afternoon a week. In Tonga, retired teachers and preschool teachers were the most commonly recruited facilitators. Potential candidates for facilitators were identified and approached by local education committees or town officers to gauge interest, explain the role and recruit to the position of volunteer community facilitator if interested. Once selected, facilitators are required to attend an initial training that provides them with knowledge and ideas on how to establish and operate a playgroup with caregivers and children. As the importance of ECD is not well understood among many communities in the Pacific, and playgroups are a new concept in many places, such training begins with developing an understanding of these concepts, and how to transfer this knowledge to caregivers and children. The length and detail of the training can vary depending on the availability of funds and the expected qualification requirements of facilitators. A minimum amount of time for the initial training would be one week, covering such topics as promoting 5 playgroups to families, facilitating groups and role modelling appropriate interactions with children for caregivers, and taking care of resources. The training is typically very practical, providing facilitators with plenty of opportunities to practice some play-based activities they can run during playgroup sessions. The hands-on nature of the training has proven to be a very engaging and thought-provoking technique. Following a period of carrying out playgroups (six months to a year), it is recommended refresher trainings be conducted for facilitators to provide an opportunity to come together to reflect on their experiences, share ideas with each other, emphasize key messages from the initial training, and learn more activities they can use in future sessions. Good practice would be to continue these refresher sessions annually, not only to develop and share new ideas, but also to build and maintain networking between facilitators, which can also help facilitators tackle challenges they face in their own communities. Mentoring/coaching facilitators Evidence shows that training is most effective when followed up with regular coaching or mentoring. This is particularly important for new recruits, who have likely been introduced to a lot of new concepts during an intensive training, and do not have the confidence to implement everything they have learned, or remember all of the details, until they develop experience over months and years. As new recruits gain more and more experience, frequency of mentor visits tapers off. Each visit by a mentor to a playgroup provides an opportunity for a facilitator to get constructive feedback on their performance, in a non-threatening, confidence-boosting manner. In doing so, the facilitator will continue to improve in their role, gain more confidence in their own ability to do the job well and to try new ideas, and also gain the confidence of caregivers and the community. However, it is critical that the mentor-facilitator relationship is clearly explained and understood by both parties, to ensure it does not become a judgmental, nerve-wracking experience. The exercise is not about assessing how well a facilitator is performing, but the mentor can highlight what a facilitator is doing well and provide some ideas to help them improve their activities or interactions with families. In Tonga, the mentoring role was initially provided by a small team of School Readiness Support (SRS) Officers located centrally in Nuku’alofa, and over the three years of the pilot intervention, expanded to include MET officers located in the outer islands to reduce travel time and costs. The SRS Officers were provided additional training in their role as mentors following a similar timeline to the facilitator training, with an initial training followed by annual refresher trainings. Monitoring and Supporting Playgroups Monitoring and supporting playgroups are also important roles to help identify changing needs or issues faced by playgroups and communities and solve problems as they arise. It requires periodical visits to playgroups (preferably multiple visits each year) to ensure in-person observations from someone outside the community of what is happening on the ground, and more regular communications with facilitators and relevant community leaders via phone/email. This requires time and effort. In small island contexts such as the Pacific where availability of human resources is thin, combining these functions with the mentoring role is one option, as was the case in Tonga and Tuvalu. In Tonga, the SRS Officers conducted site visits to all CPBAs across the country, a minimum of once 6 per school term, to provide the roles of mentor, monitor and support. During these visits, they would: provide refresher training to facilitators; take advantage of mentoring opportunities; identify and solve problems affecting CPBAs, which may include liaising with local communities, churches etc.; identify communities “at risk” of being unable to support or sustain CPBAs so that additional support could be provided to those communities/facilitators; and collect monitoring data. Developing a suite of monitoring tools is key to be able to capture as much information as possible about factors impacting the successful operation of playgroups. These include: enrollment; attendance; planning and activity records collected by facilitators; and playgroup reporting forms collected by mentors, monitors and supports. This information then needs to be fed into a database where it can be collated and analyzed. The earlier implementation monitoring mechanisms are designed, tested and used, the better for understanding what is working and what is not; however, there should be flexibility for modifying these mechanisms as needed as time goes on. In designing appropriate monitoring tools, it’s important to determine what their purpose will be, and how to ensure the monitoring data is of sufficient quality. Monitoring tools can be as simple or complex as is permitted by the available resources and capacity, but it is essential to ensure any data collected is accurate, reliable, and captures the information that is needed to understand what is working well and what needs adjusting in implementation of playgroups. A small amount of good quality data is much more valuable than a larger amount of poor quality data. The Tongan team developed the following monitoring design principles in developing their suite of monitoring tools: • Simplicity. With many of the monitoring forms being filled out by community facilitators with varying abilities and experience in monitoring, the facilitator forms should be simple and easy to use, while providing the required information. Forms were tested with facilitators and revised as needed. • Communication. To ensure the monitoring data received is reliable and complete, training on monitoring includes an in-depth understanding of the importance of filling out the forms fully and accurately. The experience in Tonga highlighted the importance of explaining this clearly during each training session (initial and refresher). Once people filling in forms understand clearly the purpose of providing complete and accurate information, they are a lot more likely to do so. • Quality assurance. Develop mechanisms to ensure data collected are complete and accurate. These could include a monitor/support checking for completeness of forms or spot-checking quality of data, and using electronic devices (such as mobile phones or tablets) to collect data. In Tonga, SRS Officers had forms electronically stored on handheld tablets using Computer Assisted Personal Interview (CAPI) technology, which provides the ability to insert data checks and error messages when unexpected or incomplete data is entered, and speeds up the data collection and reporting process. However, use of the paper and pencil could still be a back-up option in case of lack of training or other technical issues. • Unique Child/Family Identifiers. Unique child and/or family identifiers help ensure the information corresponding to each child is accurately assigned to that child, generate accurate information and data trends, and promote the reliability of results and related decision making. In Tonga, a unique child identifier was assigned to each child participating in PEARL interventions using a birth certificate or vaccination form to obtain unique information. Every child was given a unique code 7 derived from their family name, date of birth, gender and initial location (island, district, community). In theory this enabled them to be tracked across each term of the CPBA for the full period of operation of the CPBAs (maximum of 4 years). In practice, this was not necessarily foolproof, but greatly increased the likelihood of obtaining reliable and accurate data at the child level. A suite of CPBA monitoring forms were developed that were filled in by facilitators and SRS Officers, with the facilitators’ paper forms collected at each SRS Officer visit and sent to the PEARL office for electronic data entry. CAPI forms were entered electronically in real time by SRS Officer. An example of keeping things simple was the use of pin boards in the PEARL office used to keep track of each CPBA across the country. The suite of monitoring forms used in Tonga can be found in Annex 2. C. Early Grade Reading Intervention In the last 15 years, governments and evaluators have worked together to test the effectiveness of several models designed to build stronger foundational reading skills among students in the early grades of primary education. Although there are differences in the delivery mechanisms across models and the type and intensity of activities included, they follow a common approach that relies first on identifying what students can and cannot do; helping teachers and/or instructors address the skill gaps identified; monitor implementation regularly to understand what works and does not work in a particular context; and finally, reassessing students to track their progress against clearly defined learning milestones. The results of EGRAs carried out in Tonga in 2009 and again in 2014 pointed to weaknesses in children’s ability to read across all basic reading skills, due partly to problems outside of the classroom (i.e. in the home), but also to issues with the process of reading instruction inside the classroom. Further analysis found a disconnect between curriculum expectations and teaching pedagogies as they related to reading in grades 1 and 2. As a result, and early grade reading (EGR) intervention was designed to ensure full alignment between the curriculum and teaching practices, which are expected to lead to improved reading outcomes of students. Similar issues were detected in Kiribati and Tuvalu following analysis of EGRAs carried out in each country. In considering replicating a successful approach in a new country, implementors and decision makers must study how the differences in context could affect the program’s adaptation and implementation. In Tonga, the program was able to contextualize a full EGR intervention and implement it successfully mainly because of the availability of government staff that could be seconded to lead the program operations and play an integral role in the development of instructional materials and teacher coaching activities. In Kiribati and Tuvalu, due to limited funding and time remaining under PEARL at the time of the government requests, the scale of implementing interventions was not as large as in Tonga, although the same approach (as described below) was adapted to each country context. Based on rigorous evaluations of successful interventions in similar contexts2, the EGR intervention model adapted under PEARL included five elements : (1) an instructional guideline outlining 2 See Graham and Kelly, 2018. 8 the scope and sequence for teacher use during the school year; (2) supplementary instructional materials for teacher and students; (3) training for teachers to teach literacy with simplified instructions and evidence-based curricula; (4) coaching support to teachers and program implementation monitoring; and, (5) student assessment tools and the training to teachers and/or assessors on how and when to administer them. Each of these elements are critical in an EGR intervention, and failure to include any would compromise the full effectiveness of the program, and lead to poorly informed decisions around adaptation and scale up, as described below. Providing an instructional guide and training to teachers on how to use simplified instructional materials in their classroom teaching are the core elements to ensure instruction is focused on fostering predictors in an efficient (time on task) and appropriate way (modelling of the activity). Additional reading materials reinforce learning by providing targeted practice for children (decodable, short stories) and foster reading for enjoyment. Reading coaches support teachers as they implement the program during the school year, when the challenges to maintain the intervention emerge and the risk of teachers reverting to old practices becomes very real. Coaches help teachers guide pedagogical improvement and reflection, and when they also act as monitors, they gather information on program implementation and pedagogical transformation that may be useful for future iterations. Finally, the assessment tools prepared for the intervention provide the missing link between instruction and student results: at the diagnostic phase, they serve to identify skill gaps that need to be addressed. During the school year, they help teachers spot students at risk of falling behind and the skills they are struggling to master. At evaluation points, the tools help measures the effectiveness of the intervention to deliver learning expectations as per curriculum. EGR instructional and assessment resources The EGR instructional resources are at the core of the program to guide instruction, student practice and evaluation of learning through mastery checks. The ideal instructional package includes a teacher guide, supplementary readers and workbooks for students to practice decoding and fluency skills, and a set of assessment tools for teacher use to track student progress during the school year. In many countries, these resources are not readily available and will require an initial investment to be developed. While this initial investment will require some financial and human resources above the ongoing costs of implementation, it does not need to be prohibitive or require a full curriculum revision –unless the local evidence suggests so, which was not the case in Tonga, Kiribati or Tuvalu. The experience of the many countries that have already implemented EGR interventions is available free of charge to be used as reference. Expertise on how to develop these resources for the local context will be needed, as well as local linguists, educators and curriculum writers who will be best positioned to create and/or adapt EGR materials. While many books may have been donated or procured by different donor programs in PICs, some of these are not adequate for beginning readers or simply are not in a language the students know well. Production of reading materials in local language does not need to be resource intensive. Educators and government officials around the world have produced guidelines for writing decodable and short stories 9 that teachers can follow using technological innovations such as the Bloom Library3. Under PEARL, a two-year EGR program in Tonga, “Come, Let’s Write and Read” (CLRW), for Grades 1 and 2 provided a package aimed at supporting teachers to help students master the reading skills expected of them in the national curriculum by the end of grade 2. No revisions to the national curriculum were required. The package included instructional materials for teachers and students, including a teacher guide, student workbooks and readers, and assessment tools for teachers to track student progress during the school year, all in Tongan language, the language of instruction for Grades 1 and 2. The teacher guide maps out the content that must be covered in every lesson across each term of the school year. The scope and sequence for reading outlines the relative importance of different skills as each school year progresses. The daily scripted lessons are linked to the supplementary readers and student workbooks, making it easy for students to practice the content of the day and review prior lessons. The Pupil’s R apid Assessment Test (PRAT) was used to measure improvements in accuracy in letter name knowledge, sound recognition, and reading comprehension (short sentence and a one-paragraph story). The purpose of these mastery checks was to spot students falling behind and adjust the pace of instruction or introduce additional support for struggling students. All the instructional and assessment materials were produced locally in Tonga, by international and local consultants supporting officials from the MET. In Kiribati and Tuvalu, the same approach towards materials development was followed, though on a smaller scale. Staffing Arrangements for Resource Development Adapting and ultimately implementing an EGR program requires technical expertise and acute knowledge of the language and context in question. While international experts may be brought in to provide technical guidance, there is no substitute for the ownership and knowledge of local staff. Securing the continuous presence of technical and operational expertise is essential for the program activities to take place effectively. Resource development will require a dedicated group of staff who understand reading pedagogies as well as the reality of classrooms in the country of question. The initial development of materials can be done well in advance of the start of the intervention or in a staggered approach during the intervention roll–out, depending on the availability of local experts who can support the task, access to funds, and project timeline. The development of instructional materials involves the continuous support of in-country experts to complete the design, pre-test and roll-out to address technical questions and disputes that may arise. Feedback from teachers, teacher trainers and coaches will inform revision of materials, but it is the results of the student assessments that will guide any major alterations to the structure and content. Teacher Training and Coaching Teacher training requires staff who know the EGR instructional material intimately as well as the process of adult learning. Effective teacher trainers recognize the experience and wisdom in-service teachers bring to training workshops and prepare the content and delivery of the training workshop to harness it. They know the training workshop is not only an opportunity to review the breadth and depth of the instructional materials, but they must ensure teachers are provided enough opportunity to practice 3 A free, online platform for creating, translating and downloading simple books for children. https://bloomlibrary.org 10 and demonstrate to their peers. For this reason, EGR interventions under PEARL included at least two intensive trainings a year, from one week to 10 days, typically during the school holidays to avoid classroom interruption. Training was delivered directly to teachers by teacher trainers, and used a role- play approach to allow teachers to practice the required skills as much as possible during the training. At the end of training workshops, teachers were expected to leave with a clear understanding of what’s new in their practice, how to do it, and whom to ask for help when they need it. Ideally in-country language and pedagogical experts form part of the teacher training team either as teacher trainers (direct training model) or master trainers (trainer-of-trainer model). Master trainers and teacher trainers must receive explicit training and be given time to prepare for the training workshops. Coaching is a continuation of the teacher training and can be defined as providing onsite, job- embedded, sustained professional development for teachers. Through interaction with coaches, teachers get the opportunity to further improve their new pedagogical skills while reflecting on their practice and the challenges they face with their students during the school year. Changing instructional practice requires instructional coaches—people who guide and provide academic support, who can consistently do hand holding, demonstration, mentoring and monitoring of teachers. Coaching, therefore, is an imperative link in the training process to support teachers to become effective. It is important to note that a coach is different from a school inspector. While inspectors tend to focus more on compliance with the system, coaches must focus on the professional development needs of teachers to improve their practice. Staffing arrangements are essential as well as funding and creativity to find ways to ensure teacher support is delivered evenly to all teachers. In all participating countries under PEARL, the EGR program benefited greatly from the secondment of a group of senior officers from the ministries of education to plan and roll-out program operations. The program ran a staggered approach to resource development, training and coaching starting with Grade 1 materials in the first year, and Grade 2 in the second. In Tonga, year three of the pilot was devoted to evaluating the instructional, teacher training and coaching materials, and revising accordingly in preparation for scale up. There were clear advantages to having the same group of staff completing all stages of the process (materials development, training and coaching) as they gained in- depth knowledge of each stage. Monitoring and supervision of EGR program activities Since EGR programs operate as proof of concepts for potential system-level policy changes, understanding what the evidence says (and does not say) is crucial. Are the results observed in the program due to factors of design or implementation? To answer this question, program monitoring must measure implementation fidelity, i.e. was the program implemented as designed?, and document the challenges and changing needs of teachers and schools as the program progresses to draw operational insights for future iterations. In the case of EGR programs, the main vehicle driving the intervention’s impact comes from improved teacher practices. Thus, the EGR program should put in place adequate monitoring protocols to capture changes in teacher practice that can be attributed to the program, through regular classroom observations. Rather than an assessment of teacher knowledge or competence, a classroom observation tool emphasizes a set of teacher-student interactions and instructional strategies that correlate with letter 11 and word-level reading skills, and fluency. Data from classroom observations help measure teacher responsiveness to student queries and interactions, and the engagement level by students during the lesson. A good practice is to ensure that monitoring of implementation fidelity is carried out by a specialist not involved in the EGR program to maintain objectivity. If the tool is used primarily to inform coaching, it is advisable that an independent monitor assesses the teachers as well, in a separate visit, to validate the results observed by reading coaches. Staff administering a classroom observation tool receive adequate training on the goal, protocol and how to provide feedback, in addition to other tasks derived from the use of the tool. Training activities for staff administering classroom observations include assessment of inter-rater reliability4 to ensure consistency in the administration of the tool across monitors. Under PEARL, the reading coaches administered the classroom observation tool monthly during the school year and at least once per term in the most isolated schools. Although the tool was administered on paper by the reading coaches, the data collected was captured using a digital interface, the Tangerine application, designed specifically for the purposes of recording data from early grade reading and math assessments, and interviews with students, teachers and principals. The information collected helped coaches provide targeted feedback to each teacher in the program. In Tonga, the classroom observation tool was additionally administered twice a year by independent assessors. The results collected by the independent monitors validated the improvements in teacher practice observed by the program’s reading coaches. The data collected by the independent monitors produced short snapshots of teacher performance delivering the program that helped teachers, coaches and program implementers understand teacher progress across all pedagogical routines, both individually and collectively (see Teacher Profiles in Annex 8). Defining the mechanisms and approach to monitor implementation fidelity of EGR programs has important resource implications and is dependent on program objectives and context. To the extent possible, efforts should be made to rely on national or existing systems as these benefit from awareness and recognition of the purpose and delivery. However, if these do not exist, a suite of monitoring tools must be developed to capture as much information as possible that can help explain the program results at the student, teacher and school/community level. At a minimum, information about student enrollment and attendance is necessary to interpret student reading outcomes but also to understand current and future challenges to program implementation. For example, if there is chronic student absenteeism during the year, teachers may choose to move slowly across units which in turn will lower the intensity of treatment and reduce the magnitude of improvements in learning gains at the end of the school year. To better understand how system-level factors affect and react to EGR program activities, the program’s implementation data must be entered in a database for analysis. The earlier in program implementation monitoring activities are implemented, the earlier information about what works will be available to sharpen the tools and protocols. In designing monitoring tools, determining purpose and quality assurance protocols are essential to ensure reliability in their findings. Flexibility is essential to 4 The degree to which student test scores are consistent across different assessors. 12 ensure the monitoring tools adapt to the environment, not vice-versa. However, flexibility cannot compromise the accuracy and reliability of the monitoring data. In developing monitoring tools for the Tongan EGR program, the team applied the same design principles used to design playgroup monitoring tools, i.e., simplicity, quality assurance and unique identifiers for participants (see paragraph 30 above). The full suite of monitoring forms developed for the Tonga EGR program are included in Annex 7. D. Alignment of interventions In Tonga, the impact evaluation was designed to not only look at the impact of each of the two main interventions, school readiness and reading, but also to see if the school readiness intervention impacted on reading, following the ‘skills beget skills’ theory. This was done by looking at the children who were exposed to neither, one, or both interventions, to determine if the small investments in playgroups could contribute to a child’s reading abilities at the end of Grade 2. While this was difficult to ascertain clearly due to the intent-to-treat approach used for the CPBAs (i.e. communities in the treatment group could select to implement the intervention, and families could select whether or not to attend), there were noted benefits observed. Positive effects were found on reading achievement for 2nd grade girls and boys whose community was exposed to one year of the school readiness intervention and 1st grade girls exposed to two years of the community school readiness intervention. The following chapters provide further theory and evidence of the interventions carried out under PEARL. Other reports documenting the results and lessons learned have been made available online at www.worldbank.org/pearl. The last chapter offers key considerations to sustain and maximize the proven successful interventions implemented under the pilot. A classic approach to design evidence-based policy and program is to first test an innovation at smaller scale, rigorously evaluate it, and scale the pilots up in the same context if it is shown to be successful. This method has been applied by the PEARL participating governments that have created space for piloting and evaluation as a means to test promising new innovations or tweaks to existing programs to ensure good results are achieved as intended. However, there can be important differences in context even within the same country or state, so it is important to factor in administrative capacity to implement the interventions on a broader scale and to consider potential general equilibrium effects. Key elements for scaling up an evaluated pilot are included in chapter four. 13 Chapter 1 – Social and economic arguments for investing in school readiness and early literacy This chapter provides evidence-backed arguments that practitioners can use to convey the importance of investing in school readiness and early literacy to decision-makers influencing policy and financing. It highlights the social and economic arguments by demonstrating how such investments benefit individuals, families, societies, and the economy. A. Good education leads to good outcomes for individuals and societies Education is a key component of human capital, economic growth and shared prosperity. Better educated individuals have better employment, earnings, resilience, citizenship engagement and greater life satisfaction; better educated communities and countries have more economic growth, less poverty, more social mobility, better institutions and greater social cohesion (World Bank 2018). Literacy is a crucial component of human capital. Specifically, it improves individual productivity through many paths, especially in developing countries, including diffusion of technology (Basu and Foster 1998; Rosenzweig 1995). Research characterizes literacy as a necessary threshold for economic development (Azariadis and Drazen 1990). Education can be considered an investment both in terms of educational attainment and in terms of the skills acquired by individuals. A substantial body of research has examined the costs and benefits of investing in educational attainment. Psacharopoulos and Patrinos (2018) reviewed 1,129 estimates of the returns to educational attainment across 139 countries spanning 70 years and found that a year of schooling increases hourly earnings by 9 percent on average, a substantial amount which exceeds the 2.4 percent5 average returns to stocks and bonds in the United States over the past 50 years. Returns to education are higher for women and higher in developing countries (Montenegro and Patrinos 2014). An increasing body of research established empirically that cognitive skills, often measured through standardized tests, are strongly associated with future earnings. For example, in a review by Patrinos and Psacharopoulos (2010), having higher cognitive skills by one standard deviation is associated with 17 to 22 percent higher earnings on average across countries. Valerio et al. (2016) found, among countries participating in the World Bank Skills Towards Employability and Productivity survey, that having higher literacy achievement by one standard deviation corresponds to 12.6 percent higher earnings or 5.6 percent higher earnings conditional on the number of years of schooling. Despite significant increases in enrollment rates, poor learning outcomes in many countries have given rise to a “learning crisis”. For example, in the Organisation for Economic Co -operation and Development’s (OECD) Programme for International Student Assessment (PISA), the proportion of 15 - year-old students achieving minimum reading proficiency is low for middle income countries and very few attain advanced proficiency (Figure 1.1). In achieving minimum reading proficiency, students can infer information from a text, recognize the main idea of a text, and make comparisons. Students at the advanced level of reading proficiency can retrieve deeply imbedded information from and critically 5 See http://blogs.worldbank.org/education/better-returns-us-stock-market-invest-education 14 evaluate a text (OECD 2016). Similarly, low reading outcomes were found in African countries participating in the Programme d’analyse des systèmes éducatifs de la CONFEMEN (PASEC) 2014 student assessment; on average 71.4 percent of grade 2 students and 57.3 percent of grade 5 students are not proficient in literacy (PASEC 2015). Students achieving minimum reading proficiency is low for middle income countries Figure 1.1: Percent of 15 year-olds attaining minimum and advanced proficiency in PISA 2015 literacy For the Pacific Island countries, less than half of students meet the benchmark level of literacy achievement. The PILNA conducted in 2015 found that only 46.4 percent of grade 4 students and 45.7 percent of grade 6 students achieved the benchmark level of literacy proficiency for their grades (Figure 1.2). Like other regions in the world, PICs have made substantial progress increasing access to primary schooling. Improving learning outcomes presents a new challenge. Less than half of students tested in PILNA attained minimum proficiency in reading Figure 1.2: Percent of students achieving minimum reading proficiency in PILNA Source: World Bank analysis of EQAP 2016 15 B. Investment in school readiness and early literacy improves learning outcomes for children Investment in school readiness and early literacy can not only help improve educational outcomes and alleviate the learning crisis but ultimately improve human capital. The effectiveness of providing evidence-based pedagogy or community learning through playgroups has been established through rigorous evaluation methods including randomized-controlled trials. These studies establish a clear link between investing in early grade literacy and school readiness with learning outcomes, and because of the link between learning outcomes and earnings, it follows that they also strengthen human capital. Long-term studies confirm this results-chain. For example, a randomized evaluation of home visits by community nurses to encourage early stimulation and nutrition to stunted children in Jamaica yielded a 38 percent increase in the stunted children’s earnings 20 years later (Gertler et al. 2014). In addition, a randomized evaluation of providing high quality preschool services to at-risk children in the Perry/High Scope program increased participants’ median monthly earnings by 42 percent by age 40 (Schweinhart et al. 2005a). Investing early in a child’s life has a clear effect on future human capital. C. Public investment is essential Early educational interventions can significantly improve a disadvantage child’s future; public finance of these interventions is essential for promoting equity and quality of education services. There are very strong economic arguments for why public investment in education is justified if not essential. Economic arguments typically revolve around market failure. For example, a free market for education fails because children cannot borrow against future earnings to finance early education. Another is that an individual’s education not only benefits that individual but creates spillover benefits for others; better education is tied to better economic growth, institutions, and participation in governance (World Bank 2018). Yet a third is an information failure in that parents and individuals more generally have difficulty observing and understanding learning outcomes (e.g. the need to socialize learning outcome information for parents in Bruns, Filmer and Patrinos 2011). However, perhaps the most compelling reason for government financing is providing disadvantaged children with substantially higher future prospects. This is evidenced by the effects of the early stimulation intervention in Jamaica and high quality preschool intervention on disadvantaged youth’s future outcomes described above (see Gertler et al. 2014 and Schweinhart et al. 2005b). Investing early in a child’s life can dramatically change his or her future path, especially for disadvantaged children. Not only is this desirable by its own right through promoting equity and social cohesion, but it represents a safe and effective public investment because of the large body of rigorous experimental evidence showing the effectiveness of early interventions. D. Children without solid foundations start behind, and stay behind EGRAs in the Pacific between 2009 and 2013 found that children struggle with reading early on, in the first few grades of primary school. EGRA studies in Tonga in 2009 and in Vanuatu in 2011 found that after three years of schooling, only 30 percent of students in Tonga and 25 percent of students in Vanuatu were able to read fluently for comprehension (World Bank 2012a, 2012b, 2012c). In Papua New Guinea, 16 EGRAs conducted in four provinces between 2011 and 2013 found that students lagged two years behind curriculum targets for fundamental pre-reading skills (World Bank 2014a, 2014b, 2014c, 2014d). Literacy skills are persistent: children, who fall behind early, tend to fall even further behind as they age not only in literacy achievement but education outcomes more broadly. Gaps in reading skills develop early and tend to persist as children age (Butler et al. 1985). This is sometimes biblically referred to as the Matthew Effect (Stanovich 1986). Early literacy is an important determinant of a child’s future literacy as well as education outcomes more broadly including learning achievement, the likelihood of leaving school early, and transitioning to higher levels of education (Marteleto et al. 2008; Entwisle et al. 2005; Jimerson et al. 2000; Alexander et al. 1997). Figure 1.3: Differences in reading achievement by cohort Source: Good, Simmons & Smith (1998) E. Specific teaching practices and play-based learning provide a solid foundation Exposing early grade children to pedagogy that is evidence-based and context-sensitive can improve their fundamental reading skills. International research has identified pedagogic approaches to teach the skills and proficiencies that children need in order to learn how to read alphabetic languages (e.g.: August & Shanahan 2006; National Institute for Child Health and Human Development 2000; Pressley 1998; Snow, Burns & Griffin 1998). These pedagogic approaches have been adapted to difficult contexts and applied in several countries; randomized-controlled trials of these adaptations show significant improvements in early grade reading assessment scores including Uganda, Liberia, Kenya (Piper and Korda 2011; Piper, Zuilkowski and Mugenda 2014; Lucas et al. 2014; Kerwin and Thorton 2015), Papua New Guinea (Macdonald and Vu 2018) and Tonga (Macdonald et al. 2018). For example, the adaptation of evidence-based teaching methods to the Tongan context nearly doubled the percent of students classified as readers with comprehension at the end of grade 1 and increased this percent at the end of grade 2 from 18 to 29 percent (see Chapter 3). 17 In addition to high quality and relevant pedagogy at school, children need the physical, socio- emotional and cognitive develop to be ready to learn in a school environment. Research has identified that a child’s development across these domains contributed to his or her readiness to learn in school (Black et al. 2016; Nores and Barnett 2010). The lack of development across these domains has been linked to lower educational outcomes in the future (Heckman and Masterov 2007; Reynolds et al. 2001; Feinstein 2003; Pianta and McCoy 1997; Currie and Thomas 1999). Interventions targeting the home environment of young children are crucial because brain development is most malleable for young children. Cognitive and emotional function develop within the first few years of a child’s life (Young and Mustard 2007). Research has shown the effectiveness of interventions that improve early stimulation and nutrition with parent education that is delivered through education or health centers, home visits, or community groups (as in Berlinski and Schady 2015; Naudeau et al. 2011). Emerging evidence suggests that community-run playgroups offer a model for improving early stimulation of young children through the home environment. Community group learning has been shown to affect parenting behaviors first in the context of community mothers’ groups for prenatal health and child nutrition (Prost et al. 2013; O’Rourke, Howard-Grabman and Seoane 1998) and, more recently, in the context of playgroups (Brinkman et al. 2015; Nakajima et al. 2016; Macdonald et al. 2018). For example, an intervention in Tonga that offered support to communities to establish and run playgroups increased overall school readiness by 28 percent of a year of aging for disadvantaged girls (Figure 1.4). It also improved literacy and numeracy development by 22 and 26 percent of year of growth for disadvantaged boys and girls, respectively and early grade reading outcomes for children after they entered school. The effectiveness of peer learning in this intervention built on the strength of communities in the Pacific more generally (Farran 2009; Toganivalu 2008; Huffer 2006; Griffen 2006; World Bank 2013). 18 Being in a community that received support to establish a playgroup increased school readiness outcome for disadvantaged children Figure 1.4: Effect of the school readiness intervention as a % of developmental gains between age 3 and 4 Source: World Bank 2019 F. Conclusion Investments in early childhood development are shown to have the greatest impact on children’s wellbeing and lifelong learning, with benefits reaching beyond the future economic prospects of the individual and family to broader positive impacts on communities, societies and the economy. The PEARL program piloted two interventions focused on improving school readiness and early literacy in the Pacific context and found positive benefits on children’s overall development and reading with comprehension. Lessons from these interventions are detailed in Chapters 2 and 3, as well as why and how these specific interventions were selected, designed and adapted for PICs. 19 Chapter 2 – Designing Interventions to Promote School Readiness This chapter focuses on interventions aimed at improving the school readiness of children, drawing on examples and lessons from the Pacific under the PEARL program, and how school readiness outcomes can be monitored and measured to improve policies and programs. A. How to identify interventions to improve school readiness The process for selecting school readiness interventions appropriate for the local context draws on multiple sources of information obtained locally, regionally and globally. While it is important to apply rigorous proven successful interventions in similar contexts, it is in fact more important to adapt those based on local data and evidence. Local data provide a valuable source of information for understanding what is currently happening at the country and community level, what are local perceptions of school readiness and ECD, and what may or may not work in the local context based on local policies and practices, and available resources. Regional data provide information on what else is happening in similar contexts, while global data draws on international literature to better understand child development, and provides information about what does and doesn’t work in various settings around the globe. Tonga and Tuvalu presented two different cases of what was happening in each country in the space of early childhood development prior to the introduction of PEARL interventions, most notably the 95 percent (in 2016) coverage of preschools in Tuvalu compared to an estimated 25 percent (in 2014) in Tonga. In order to design interventions that were most suitable for each context, several studies were conducted in each country to understand what could be piloted, including community dialogues, policy and program assessments, and early childhood development surveys at the population level. The results for both countries led to the conclusion to pilot playgroup activities for parents and children; however, the reasons for implementing them and the ways they were implemented were different, based on what was happening in each country at the time. What works to promote early child development and school readiness outcomes? Globally, research has demonstrated that with the appropriate policies, programs, and supports, children’s development and readiness for school can be improved (Britto et al. 2016). Encouragingly, investments in children’s early years have been demonstrated to have as much, if not a more positive influence on socioeconomically disadvantaged children when compared to their more advantaged peers (Elango et al. 2015). In this way, it can be argued that through working to promote development in the early years, the negative impacts of disadvantage can be countered and inequality in children’s outcomes reduced. From a developmental perspective, two key types of modifiable factors or interventions in the early years have been consistently demonstrated to improve children’s school readiness outcomes across a variety of cultures and contexts. The first is providing children with stimulating, responsive and nurturing home environments (see for example Britto et al. 2016; Melhuish et al. 2008; Rao, Sun, Chen, and Ip 2017). This includes aspects of cognitive (e.g. reading, storytelling, counting, and drawing, etc.) and socio- 20 emotional caregiving (singing songs, playing games, taking the child outside or away from home, etc.), responsiveness to children’s needs (e.g. health and safety needs, or communication needs) as well as a loving, warm and secure environment (e.g. a secure parental attachment). Recent reviews of interventions aiming to promote and improve these types of interactions between parents and their children have demonstrated positive effects on children’s cognitive, non-cognitive, and physical development (Britto et al. 2016). Second, children’s participation in both formal and non-formal types of ECD - for example preschool, community playgroup, and child care - have consistently been demonstrated to improve children’s readiness for the school environment (Domitrovich et al. 2013; Duncan and Magnuson 2013; Gregory et al. 2016). Research shows that children who do not have the opportunity to attend ECE before school are more likely to find the transition to the school environment challenging and are at a higher risk of early drop-out and school failure. As a result of the evidence base, combined with the fact that unfortunately millions of children miss out on these critical opportunities for early stimulation globally, interventions aimed to improve children’s developmental outcomes often target one, if not both of these key factors. Interventions that address gaps in the provision of opportunities for children to develop are important responsibilities for governments and communities to consider, and require an understanding of the local context as well as what works for school readiness locally, regionally and globally. Given the importance of a child’s early environment for school readiness, parenting or caregiver practices are crucial to promoting school readiness; the challenge is how to influence parenting practices. The very first people that play a key role in a child’s development are the parents and/or caregivers of a child, as they have the most contact and opportunity to interact with their children in the early years. If parents and caregivers are not aware of or do not understand what is important for child development, or even why these daily interactions with their children are so important, then interventions that educate them and provide hands on experiences engaging with their children are shown to be effective (Britto et al. 2016). Such interventions can be led or guided by health professionals, teachers, social workers, or other professionals with experience in and an understanding of child development. Home visits (e.g. in Jamaica, Bangladesh and Brazil) by visitors trained in child development and the defined intervention6 have shown positive outcomes on parenting behavior. These visits provide parents and caregivers with knowledge and practical experience in looking after and interacting with their children. These types of interventions are one-on-one, are costlier than group interventions, and are usually provided by the State. They also target more vulnerable families ensuring that the most needed receive the services. These have been shown to lead to positive gains towards improved child development outcomes on various domains. Another approach is community learning including women’s groups or community playgroups. For example, community groups have been used in the health and nutrition to affect parenting behaviors including community women’s groups to promote feeding practices among young mothers (Prost et al. 2013; O’Rourke, Howard-Grabman, and Seoane 1998). In Australia, playgroups led by trained early 6 Level of required qualifications and training for home visitors varies across countries. 21 childhood development practitioners provide caregivers and children with interactive activities that stimulate the development and learning of the child, while providing the caregivers with ideas and knowledge of what’s important for children at various stages of their development, and what they can do to promote good child development. They are often community-led, and low-cost, also providing a space for caregivers to interact with each other, discuss their own caregiving practices and concerns, and get ideas from each other. Britto et al. (2016) found a combination of home visits and group sessions to be more effective than home visits alone. In Tonga, community playgroups increased parent interaction with children conducive to school readiness. A school readiness survey conducted in 2017 in Tonga found that children who attended community playgroups had parents that were more likely to read books, tell stories, sing songs, play with their children and name, count or draw things with their children (Figure 2.1). An impact evaluation found that parents in communities that received support to establish playgroups were also more likely to do many of these activities with their children as well (Macdonald et al. 2018). The Tonga school readiness intervention increased caregiver interaction with children on cognitive domains Figure 2.1: Influence of playgroup attendance on home learning activities Source: Macdonald et al. (2018) A majority of countries around the world provide preschools for children for one or two years prior to starting primary school, though there are large variations in access to preschool due to financial constraints, quality of preschools, and public perceptions around the importance of preschool. Preschools ideally provide children with a play-based curriculum that teaches children an array of important skills to prepare them for primary school, including pre-literacy, pre-numeracy, gross and fine motor skills, working in groups, listening and taking turns, emotional control, among other important school readiness skills. In many Pacific countries, preschools are often provided privately, leading to costs for parents that can in many cases be unaffordable. 22 Assessing governance, institutional and policy capacity to promote school readiness A first step in identifying interventions to improve school readiness is an assessment of the capacity of government to intervene and promote school readiness. The World Bank’s SABER-ECD tool for example provides an in-depth analysis of the enabling environment of various aspects of ECD in a country. This instrument collects data on national level policies and provides a method to analyze the quality of policies and design of programs for (1) establishing an enabling environment, (2) implementing widely, and (3) monitoring and assuring quality. The results of the SABER-ECD tool help to better understand the strengths and weaknesses of government policies, legislation and activities towards promoting early childhood development and school readiness. It includes consideration of early learning, health, nutrition and social and child protection areas alongside regional and international comparisons. It is normally undertaken by a local person with an in-depth knowledge and understanding of ECD in the surveyed country, working closely with the Bank’s SABER team who guide the local expert, analyze the results and develop a report. The survey includes: • international and regional comparisons of key indicators (e.g. child mortality, preschool enrolment rates etc.); • a desk review of available government policy documents, data and literature; and • interviews with a range of ECD stakeholders, including government officials, service providers, civil society, development partners and scholars. The survey works from a checklist of the types of policies, services and activities that can be used to promote coordinated and integrated ECD interventions for young children and their families. For example, it checks to see if there are standard health screenings for pregnant women and childhood immunizations (healthcare); breastfeeding promotion and iron fortifications (nutrition); and parenting programs and free preschool (early learning). Since 2012, the World Bank, working closely with other development partners including the United Nations Children's Fund has carried out the SABER-ECD survey in multiple countries in the Pacific7. Some common themes that appeared across these countries were that they lacked the capacity to: (1) measure and monitor school readiness, (2) prioritize resources across different ministries charged with the wellbeing of young children in different developmental domains, and (3) enforce policies and regulations at the ECD service provider level such as ensure preschool quality. The surveys also found that most countries in the Pacific were weak in early childhood multisector collaboration and coordination between ministries. 7 The Pacific countries surveyed between 2013-2016 include Kiribati, Samoa, Solomon Islands, Tonga, Tuvalu and Vanuatu. 23 Assessing community capacity to promote school readiness Communities have the potential to play an important role in promoting school readiness and ECD outcomes more generally as a channel to influence the home environment and parenting practices. Given the crucial role of the early home environment for ECD outcomes, parenting behavior is an important channel for improving school readiness outcomes. However, parenting behavior is extremely difficult to influence given the private nature of households but also the need to shape how individuals value school readiness. As discussed above, community interventions have been successful in shaping parenting behavior. These originally began with community women’s groups to provide training on feeding practices for young children, and emerging evidence show that community run playgroups have also been effective. Leveraging community social capital requires assessing community capacity and barriers to promoting school readiness. For example, the World Bank carried out a series of community dialogues in Tonga, Samoa and Vanuatu to understand what role communities could play in promoting school readiness and what the possible barriers could be. The dialogues were structured as workshops following a methodology based on participatory monitoring and evaluation approaches (World Bank 2013) which aimed to strengthen a community’s ability to utilize its social capital. The 2013 World Bank Pacific Island community dialogue series (World Bank 2013) revealed strong potential for communities to influence parents to better promote school readiness. The workshops held in communities found that community members generally did not believe that many parents fully understood the value of preparing children for school at a young age. However, they also believed that leaders and members of parents’ own communities were best placed to influence parents and help them fully understand the value of school readiness activities. These findings helped motivate the piloting of community playgroups in Tonga. Measurement of Early Childhood Development (ECD) outcomes Measurement of ECD outcomes is essential to designing, monitoring and evaluating school readiness policies and programs. The SABER-ECD studies in several Pacific countries had found that outcome measurement especially across the cognitive, language and social developmental domains was missing (Figure 2.2). In addition, indicators about disparity in accessing programs were not available in countries that had not participated in a major household survey including the Demographic and Health Surveys or the Multiple Indicator Cluster Surveys. Prior to PEARL, these surveys provided measures of access to services and physical development outcomes and enabled comparing results across different socio-economic sub-populations. 24 In all three countries surveyed, ECD outcome data was missing on most domains Figure 2.2: Summary of SABER instruments’ review of data availability Source: World Bank (2013) Survey data on early childhood development outcomes and what is impacting on these results provide useful information to understand what interventions may be most likely to provide improved outcomes, identify key elements of design of interventions, and improve implementation if conducted after they have been operating for some time. ECD surveys of preschool-aged children conducted in Tonga (2014) and Tuvalu (2016) using the early human capability index (eHCI) instruments were valuable sources of information in this regard, as they helped to identify the domains of child development that were weakest (disaggregated by location), what was happening inside the home in terms of interactions between young children and their caregivers and older siblings, and availability of learning materials in the home. Figure 2.3 shows results from the 2014 Tonga early human capability index (TeHCI) survey indicating the factors that had the greatest impact on the literacy domain. 25 Being read to at home has a bigger impact on literacy than attending preschool in Tonga, but combining the two has the greatest impact Figure 2.3: Factors impacting the literacy domain, TeHCI 2014 0.9 Preschool but No reading, no 0.8 Read to 0.7 Read to but NO and 0.6 attended Literacy 0.5 0.4 0.3 0.2 0.1 0 2 3 4 5 6 Age of Children Source: Baseline TeHCI 2014 Key considerations for choosing school readiness interventions Determining which interventions to pilot depends on (i) international evidence of its effectiveness, (ii) suitability to the local context including capacity, (iii) understanding where the greatest issues lie, and (iv) financial sustainability if scaled up. The international research discussed above presents insights into which types of interventions have been effective to improve school readiness, especially interventions that affect parenting practices. Assessing the institutional capacity and community capacity helps identify where interventions are needed and what may be potentially effective. Local data on ECD outcomes gives insights into where the issues are that need to be addressed. Finally, given the restricted budget of many developing countries, it is clear that low cost interventions are usually needed. Box 2.1 describes the experiences in Tonga and Tuvalu of choosing appropriate school readiness interventions. 26 BOX 2.1 SCHOOL READINESS - WHAT WAS IMPLEMENTED UNDER PEARL? Tonga: Beginning in 2015 three school readiness interventions were implemented: 1. Community Play Based Activities (CPBAs), or playgroups, in around one-third of Tongan communities. With technical advice and support from PEARL, CPBAs were established by Local Education Committees under the guidance of Town Officers and run by local parents trained as facilitators. CPBAs were implemented flexibly, in places and at times that suited the community best. Some were held in primary school classrooms, others in community halls and private houses. They were held once or twice a week, with all children aged 0-5 in the community and their parents/caregivers invited to attend. Facilitators were trained in how to organize and run a CPBA and given a starter pack each of resources for the activities. Each year from 2015-2017, around 1,700 children attended them in around 40 communities. Since PEARL support ended, many CPBAs have continued with support from communities and churches recognizing the importance of guiding caregivers on engaging in stimulating interactions with their children. 2. National Public Awareness Campaign. The main public awareness activity was the “Read with Your Child” national campaign in 2017. This was a six-week campaign using primarily TV and radio, with material also distributed through Facebook. Key aspects of the messages were that reading with your child strengthens family bonds and improves children’s school readiness. Messages were aimed at men and women, and showed people from all walks of life and ages, reading with their children or grandchildren in many different locations around Tonga. Surveys established that there was a high level of recognition of the campaign and its message. 3. Support for the Early Childhood Development and Education (ECDE) Advisory Council. Prior to PEARL, the ECE Council was the main formal ECE coordination mechanism in Tonga containing representatives of the MET, Tonga Preschool Association and non-government preschool providers. As school readiness goes well beyond preschool, this activity created an ECDE Advisory Council which was the ECE Council augmented with other Ministries and stakeholders, to operate as a reference committee for the Minister of Education and for PEARL. This process included the development of operational procedures, and assisted MET engagement with its partners, including other government agencies and non-government preschool providers. Tuvalu: Playgroups@Preschools, implemented in 2017: Playgroups were established at all 18 preschools in Tuvalu. Two preschool teachers from each preschool were trained in May 2017 and each playgroup was provided with a starter pack of resources and ideas for activities. They started well with many 0-2 year olds attending them as well as the preschool children. There has been some reduction in momentum in subsequent years showing the need for continued support, ideas and training. Under the PEARL program, as limited budget is a common issue for all participating countries, activities had to be low cost, simple and practical, culturally relevant and sustainable. Both Tonga and Tuvalu had limited budgets for early childhood education and had limited ECE staff capacity in their ministries of education. These considerations, plus the fact that many caregivers were not aware of the importance of ECE/ECD, meant sustainability would be an issue if interventions were high cost or complicated. 27 Particularly in Tonga, the results of both the SABER-ECD assessment and the community dialogues suggested the focus should be on increasing parent and community awareness of ECD. This was chosen over supporting preschools directly for several reasons. First, while participation in preschools was not high (only 25 percent of children attended), Tonga’s major development partners (Australia and New Zealand) were strongly supporting preschool, providing funding and books, development of a new preschool curriculum, including training of preschool teachers in the new curriculum. Furthermore, it was believed that implementation of CPBAs was likely to increase interest of parents in ECD and preschools, thus leading to increased demand for, and participation in, preschools. Second, the “Read With Your Child” campaign came directly out of the results of the 2014 TeHCI which showed that many children were not read to, and that children whose parents read to them were more school ready than other children. Third, support for the ECED Advisory Council came from the recognition that ECE was mainly delivered by non-government organizations – churches and communities – and that improvement in quality, participation and outcomes would only occur if the government worked effectively in partnership with these organizations, and with other government ministries. In Tuvalu, the decision was made jointly by the government with technical assistance from the Bank to create playgroups@preschools as the situation was quite different from that in Tonga. In Tuvalu, preschool enrolment was already 95 percent. However, the Tuvalu baseline of the early human capability index (Tu-eHCI 2016) results showed preschools were not having the expected impact on school readiness, which could be attributed to low attendance and/or poor quality of preschools. The government had seen the CPBAs in Tonga and were interested in piloting them in the existing preschools with their preschool teachers providing the facilitating role. This would be very straightforward to implement, would not require additional venues or facilitators, and be manageable for existing ministry ECE staff. Preschool teachers have been trained to be facilitators in organizing with parent and facilitating activities of playgroup sessions. The government plans to implement more parental awareness activities, and some are being developed as part of the Tuvalu early learning roadmap (an implementation plan for the next five years). Deciding on the scale, phasing and number of interventions The next step after considering one or more interventions that meet the key considerations are questions of scale, phasing and the possibility of comparing different options to inform future scale up investments and plans. More expensive interventions, such as home visits, are normally targeted at the more vulnerable families or communities, but even cheaper interventions are usually piloted in select communities before scaling up to larger populations. As indicated, funding and resource availability and the program’s sustainability are key determinants in assessing these considerations, as gradual scale up allows time to test and make changes based on monitoring and feedback, time for key resources to be developed and produced in greater quantities, for training programs to be developed and provided to larger numbers of participants, and for monitoring, support and supervision systems to be developed, strengthened and scaled up. Further considerations are the extent to which interventions feed into existing structures, as this will determine how much effort and resources are needed to accommodate the interventions. Tonga presented a unique case, in which the government was interested in piloting an innovative 28 approach to ECD and early literacy by developing a set of interventions that addressed both school readiness and early literacy8, and designing an impact evaluation that examined cohorts that partook in each or both interventions, and compared them to children that received neither intervention. Designing and implementing such a program with a rigorous impact evaluation requires more thorough design and adds more complexity to implementation compared to traditional qualitative evaluations. It also requires high levels of technical expertise, which may not be available in low-capacity contexts such as the Pacific. However, the PEARL program provided the Government of Tonga with an opportunity to pilot this approach, enabling the government to measure the program’s effects and identify evidence-based solutions to address the most pressing concerns of school readiness and early literacy. Monitoring systems were developed to modify interventions as required during the three-year pilots, and baseline and endline assessments of school readiness and early grade reading provided evidence of what works, enabling the government to scale up and institutionalize the successful pilots. With Tuvalu’s tiny population, scale was less of a concern, and the Ministry of Education, Youth and Sports (MEYS) with support from PEARL was able to provide training and resources to two preschool teachers from all 18 preschools in the country to enable them to implement playgroups in all preschools following a similar play-based model to the one used in Tonga. Some parents also attended one of the facilitator training workshops, which helped to build support for the playgroups. This model was effective in being able to build on existing resources in the country, including the expertise of the preschool teachers, and venues and materials of the preschools. It also meant monitoring, supervision and support to the playgroups would fit into the existing government systems with little additional effort, cost or resources. This also has an impact on sustainability, as interventions that require less additional effort, cost and resources have a greater likelihood of continuing for longer. B. Designing and implementing a multi-level intervention for school readiness In order to improve outcomes at the individual level and ensure interventions are successful and sustainable, it requires ensuring there are supports in place at multiple levels from the child and family level to the community level and the system or policy level. For the school readiness interventions targeted at children and their families described below, support and awareness raising activities were also provided at the community level where the interventions were operating, and at the system or policy level to provide greater support and sustainability. This section describes the playgroup-model of school readiness interventions, and covers the supports required at the community and system level required to ensure sustainability of the playgroups and possible expansion beyond the life of the pilot. Child/Family-level intervention: Playgroups Playgroups consist of regular (minimum weekly) gatherings of caregivers with their children under school age (typically 10-15 families) coming together to interact and engage in play-based learning activities that provide a stimulating environment for children, an opportunity for caregivers to interact with their children in this manner, and a learning opportunity for caregivers to continue engaging with their children daily. The gatherings, typically lasting two hours, are facilitated by individuals trained to run 8 The early literacy interventions are discussed in greater detail in Chapter 3. 29 the sessions. These could be parents, community members, teachers, or anyone else with an interest in early childhood development. They are run in a cooperative spirit, with parents involved in decisions on frequency of sessions, rules, and types of activities. They are built on the proposition that early childhood programs are most effective when they support parents’ and other caregivers’ active participation in the children’s early learning and development. That is, parents are provided with ideas about activities they can do at home with their children, thus increasing the exposure of the children to development activities, while also increasing bonding time between caregivers and children. Playgroups are a low-cost intervention, and around the world they are known as being suited to the needs of parents and younger children and can be tailored to meet the needs of different communities. Costs of establishing and operating playgroups can be kept low by recruiting volunteers to facilitate sessions, using existing venues to operate rather than a purpose-built facility, and using local materials for play and learning activities. Playgroups harness community strength and build on existing support for ECD. In Tonga, each community playgroup - known locally as a community play-based activity (CPBA) groups - was facilitated by a volunteer community member and supported by a village early childhood education committee or a local church. Tongan communities and community-based organizations including churches have long been the financers of preschools, with financing provided by tuition from parents, donations from community members (including those who had emigrated), community volunteers, and donations from church groups. Building on this existing support for ECD was an important factor in being able to establish and operate CPBAs in Tonga. Key considerations for the design of playgroups are: • local context and understanding of concepts introduced and discussed in activities; • local availability of materials used in activities; • available budget for activities; • local capacity for implementation; and • venue requirements. In Tonga, an international expert in playgroup design, setup and operation worked with communities on the design of the CPBAs. The design was based on those implemented successfully in isolated communities elsewhere and adapted to the Tongan context to ensure local appropriateness. A training program was developed for and delivered to volunteer, follow up trainings was also provided to reinforce concepts and share new ideas. Resources, Information and Ideas Availability of funding or lack thereof can have large implications for the kinds of activities that are engaged in during playgroup sessions, though one of the benefits of playgroups is the wide variety of interesting and engaging activities that can be carried out with a very small budget. As much as possible, local resources can be gathered, such as sand and other items found in nature, old unwanted household 30 materials, etc., which not only reduces costs, but also ensures children are able to easily identify with the materials and activities they are engaging in. It may require some costs in the beginning to get started (though these do not need to be expensive either), but the main focus of identifying resources requirements is on play-based interactions with young children, allowing them to interact with other children, adults and the environment around them in a way that stimulates their senses. It’s also important they are suitable for young children, such as not containing choking hazards or toxic substances. Under PEARL, the program also developed a ‘starter pack’ for all playgroups and facilitators, which included a variety of materials sourced internationally and locally, such as playdough, beading kits, building blocks, stationery. Activities were also designed to make use of items that can be found in the local supermarket. The starter kit included: beading and lacing kits, building blocks, toy cars, bubble sets, playdough, children’s books in local language, a variety of stationery items, a set of recipe cards to make new materials and activities (eg. for bubbles, goop, playdough, finger paint), groceries procured locally to make the recipes, and a set of fact sheets (Annex 1) for facilitators on how to run CPBAs. In addition to this starter kit, many communities provided additional materials, including tables, chairs, mats, cupboards etc as part of the venues, playground equipment, and additional materials for the play-based activities The fact sheets and recipe cards, together with the initial training of facilitators and the regular visits by School Readiness Support officers, were important ways of supplying information and ideas to facilitators, who in turn conveyed messages to parents during CPBA sessions. This was an essential part of PEARL – most people in Tonga had never seen or participated in a playgroup, and the idea that there are lots of different activities that parents can do with children to improve their development was relatively new. In many PICs, one of the biggest constraints to learning to read in mother tongue is lack of availability of reading materials in local languages. Therefore, some external support may be required to provide and produce reading materials in local languages for caregivers to read with their young children. It is important to use existing relevant materials in local language and develop new materials and make them available to users. In Tonga, MET staff developed Tongan language reading books that were printed under PEARL, and additional books were sourced from University of the South Pacific in Fiji. Sets of these books were distributed to every CPBA. Playgroup facilitators The process for selecting facilitators will require consensus from the stakeholders in consultation with local communities. Funds may exist for a stipend or salary, or facilitators may be unpaid volunteers. Irrespective of these conditions defined by the community, broadly speaking, facilitators should be committed to and passionate about working with families and particularly children, dedicated enough to commit to a regular schedule, be open to new ideas, and be prepared to undertake training in facilitating the groups. Teachers (retired, part-time or preschool), health workers (such as community nurses), and social or community workers are often good choices for facilitators, as they have the experience and desire to work with families and help them towards better caregiving practices and child development outcomes. In Tuvalu, the facilitator role became an extension of the preschool teacher’s role, opening their preschool to caregivers and younger children at least one afternoon a week. In Tonga, retired teachers and preschool teachers were the most commonly recruited facilitators. Potential candidates for 31 facilitators were identified and approached by local education committees or town officers to gauge interest, explain the role and recruit to the position of volunteer community facilitator if interested. Once selected, facilitators are required to attend an initial training that provides them with knowledge and ideas on how to establish and operate a playgroup with caregivers and children. As the importance of ECD is not well understood among many communities in the Pacific, and playgroups are a new concept in many places, such training begins with developing an understanding of these concepts, and how to transfer much of this knowledge to caregivers and children. The length and detail of the training can vary depending on the availability of funds and the expected qualification requirements of facilitators. A minimum amount of time for the initial training would be one week, covering the topics listed below. The training is typically very practical, providing facilitators with plenty of opportunities to practice some play-based activities they can run during playgroup sessions. The hands-on nature of the training has proven to be a very engaging and thought-provoking technique. Training topics include: • understanding the importance of ECD, how playgroups fit into this, and the role of the facilitator; • identifying and establishing a suitable play space in the venue; • developing a weekly schedule for playgroups; • promoting playgroups to families; • developing guidelines for the playgroup; • planning activities for each session; • facilitating groups and role modelling appropriate interactions with children for caregivers; • taking care of resources; • liaising with other stakeholders, such as preschools and schools, community health workers, community leaders, churches etc.; and • monitoring playgroups – registering families and recording attendance and activities. The longer the initial training, the greater the opportunity for facilitators to learn, understand and practice fundamental and more advanced concepts of childhood development and working with families and communities, and in doing so, developing greater confidence in their ability to do so. In some communities, families may be more inclined to participate in playgroups if they know facilitators have a certain minimum qualification in pedagogy. In Tonga and Tuvalu, many facilitators were retired teachers or preschool teachers, so the one-week training provided was deemed sufficient for them to carry out playgroups. Following a period of time of carrying out playgroups (six months to a year), it is recommended refresher trainings be conducted for facilitators to provide an opportunity to come together to reflect on their experiences, share ideas with each other, emphasize key messages from the initial training, and learn more activities they can use in future sessions. Good practice would be to continue these refresher 32 sessions annually, not only to develop and share new ideas, but also to build and maintain networking between facilitators, which can also help facilitators tackle challenges they face in their own communities. Playgroup Venues Identifying the requirements for suitable venues is important, as there are likely to be different options available across communities, but not all will be suitable for such activities. It’s key to understand what the requirements of the venue are, such as a safe, weather proof structure with sanitation facilities, and preferably a suitable outdoor space for children to play. The venue should be available for use by the playgroup as per a regular agreed schedule, without being asked to regularly change date or time, or move elsewhere. If such requests occur often, it can be difficult to retain interest from families. Ideally the venue would be made available free-of-charge, but if a cost is associated to use it, it is important to identify a sustainable source of funding to cover the ongoing rental cost. The responsibility for providing a suitable venue for the playgroups to run will be a collaboration between stakeholders, including government, churches and communities. Having identified the minimum requirements for a suitable venue during the design phase, these will be provided to relevant stakeholders to help select a venue in each community. In Tonga, venues included community halls, church halls, primary schools and private residences, while in Tuvalu they were held in preschools. Mentoring/coaching facilitators Evidence shows that training is most effective when followed up with regular coaching or mentoring. This is particularly important for new recruits, who have likely been introduced to a lot of new concepts during an intensive training, and do not have the confidence to implement everything they have learned, or remember all of the details, until they develop experience over months and years. As new recruits gain more and more experience, frequency of mentor visits tapers off. Each visit by a mentor to a playgroup provides an opportunity for a facilitator to get constructive feedback on their performance, in a non-threatening, hopefully confidence-boosting manner. In doing so, the facilitator will continue to improve in their role, gain more confidence in their own ability to do the job well and to try new ideas, and also gain the confidence of caregivers and the community. However, it is critical that the mentor-facilitator relationship is clearly explained and understood by both parties, to ensure it does not become a judgmental, nerve-wracking experience. The exercise is not about assessing how well a facilitator is performing, but the mentor can highlight what a facilitator is doing well and provide some ideas to help them improve their activities or interactions with families. In Tonga, the mentoring role was initially provided by a small team of School Readiness Support (SRS) Officers located centrally in Nuku’alofa, and over the three years of the pilot intervention, expanded to include MET officers located in the outer islands to reduce travel time and costs. The SRS Officers were provided additional training in their role as mentors following a similar timeline to the facilitator training, with an initial training followed by annual refresher trainings. Coordination and management A key feature of any successful intervention is dedicated leadership and management, with clear roles and responsibilities. In Tonga, the local PEARL coordinator had day to day responsibility for 33 developing workplans, monitoring progress, allocating tasks to team members and above all, solving problems. She advised the Ministry and World Bank management of new risks and problems as well as possible improvements in implementation. For example, through discussions with the Ministry of Internal Affairs and District Officers, she identified that rather than engaging the target CPBA communities through District Officers, it would be quicker and simpler to engage directly with town officers. Such dedicated leadership is essential, particularly when implementing something new, with many attendant risks. Monitoring and Supporting Playgroups Monitoring and supporting playgroups are also important roles to help identify what is working and what is not, identify changing needs or issues faced by playgroups and communities, and solve problems as they arise. It requires periodical visits to playgroups (preferably multiple visits each year) to ensure in-person observations from someone outside the community of what is happening on the ground, and more regular communications with facilitators and relevant community leaders via phone/email. This requires time and effort. In small island contexts such as the Pacific where availability of human resources is thin, combining these functions with the mentoring role is possible, as was the case in Tonga and Tuvalu. In Tonga, the SRS Officers conducted site visits to all CPBAs across the country, a minimum of once per school term, to provide the role of mentor, monitor and support. During these visits, they would: provide refresher training to facilitators and mentoring opportunities; identify and solve problems affecting CPBAs, which may include liaising with local communities, churches etc.; identify communities “at risk” of being unable to support or sustain CPBAs so that additional support could be provided to those communities/facilitators; and collect monitoring data. In the later years of PEARL this important role was continued by the local MET officers. There are several feedback loops that can be set up to improve implementation of playgroups. These include monitoring and survey data, mentor/support reports following visits, facilitator feedback (by email, phone etc. or during refresher training sessions), engagements with communities and churches. Developing a suite of monitoring tools is key to be able to capture as much information as possible about factors impacting the successful operation of playgroups. These include: enrollment; attendance; planning and activity records collected by facilitators; and playgroup reporting forms collected by mentors, monitors and supports. This information then needs to feed into a database where it can be collated and analyzed. In Tonga, under the pilot period, data analysis was done by World Bank staff. It was used to fine tune implementation and facilitator training. The earlier in implementation monitoring mechanisms are designed, tested and used, the better for understanding what is working and what is not; however, there should be flexibility for modifying these mechanisms as needed as time goes on. In designing appropriate monitoring tools, it’s important to determine what their purpose will be, and how to ensure the monitoring data is of sufficient quality. Monitoring tools can be as simple or complex as is permitted by the available resources and capacity, but the key is to ensure any data collected is accurate, reliable, and captures the information that is needed to understand what is working well and what needs adjusting in implementation of playgroups. The Tongan team developed the following monitoring design principles in developing their suite of monitoring tools: • Simplicity. With many of the monitoring forms being filled out by community facilitators with varying 34 abilities and experience in monitoring, it was considered important to keep the facilitator forms simple and easy to use, while providing the required information. Forms were tested with facilitators and revised as needed. • Communication. To ensure the monitoring data received is reliable and complete, it is critically important that part of the training on monitoring include an in depth understanding of the importance of filling out the forms fully and accurately. The experience in Tonga highlighted the importance of explaining this very clearly during each training sessions (initial and refresher). Once people filling in forms understand clearly the purpose of providing complete and accurate information, they are a lot more likely to do so. • Quality assurance. Developing mechanisms to ensure data collected is complete and accurate is a priority. These could include a monitor/support checking completeness of forms or spot-checking quality of data, and using electronic devices (such as mobile phones or tablets) to collect data. In Tonga, SRS Officers had some of the forms electronically stored on handheld tablets using Computer Assisted Personal Interview (CAPI) technology, which provides the ability to insert data checks and error messages when unexpected or incomplete data is entered, and speeds up the data collection and reporting process. However, use of the paper and pencil could still be a back-up option in case of lack training or other technical issues. • Unique Child/Family Identifiers. To ensure accurateness of information at the child/family level, unique identifiers are imperative. Without a mechanism to ensure the information corresponding to each child was accurately assigned to that child, it could lead to false information and trends, which impact the reliability of results and related decision making. In Tonga, a unique child identifier was assigned to each child participating in PEARL interventions using a birth certificate or vaccination form to obtain unique information. Every child was given a unique code derived from their family name, date of birth, gender and initial location (island, district, community). In theory this enabled them to be tracked across each term of the CPBA for the full period of operation of the CPBAs (maximum of 4 years). In practice, this was not necessarily foolproof, but greatly increased the likelihood of obtaining reliable and accurate data at the child level. A suite of CPBA monitoring forms were developed that were filled in by facilitators and SRS Officers, with the facilitators’ paper forms collected at each SRS Officer visit and sent to the PEARL office for electronic data entry. CAPI forms relating to the support visit and community surveys were entered electronically in real time by SRS Officer. An example of keeping things simple was the use of pin boards in the PEARL office used to keep track of each CPBA across the country (see images below). The suite of monitoring forms used in Tonga can be found in Annex 2. 35 Monitoring CPBAs in Tonga using pinboards (Photos: Wendy Jarvie) Community-level interventions: National Awareness Campaigns The rationale Communication and public awareness programs are too often an afterthought in implementing new initiatives. All efforts are making community engagement a key factor for success in implementing new project/program. The tools used to communicate, and the investment made in shifting human behavior and social norms at the community level to attract positive action and support, must be given the same level of consideration currently reserved for science and management programs. Despite growing evidence that communications have a strong impact on success and sustainability in initiating new projects, many do not allocate sufficient resources or capacity to implement effective outreach. As noted earlier, there is widespread consensus that parent and community awareness of ECE is weak in the Pacific, and a significant barrier to improving children’s school readiness. Many parents see no value in preschool (World Bank 2013, and later discussion), and do not support their local preschool. In most PICs, there are few playgroups or other community ECD activities. This is particularly true of male adults, such as fathers and grandfathers, with surveys showing few men read to or interact with children. This issue is further exacerbated in countries where there are few books and other resources in local languages for preschool and younger children, so that even if parents wanted to read to their children there is little material that can be used in the local language. This is the case in many countries in the Pacific. Again too, there is usually little support for the development needs of younger children, apart from health support provided through ministries of health. Yet the international evidence shows that highly disadvantaged children will not be ready for school without earlier interventions and parenting support. Changing awareness, attitudes and values, can be slow. Creating change at the community level requires more than simply transmitting and receiving information; it requires figuring out what moves people on an emotional and practical level and then designing communications that speak to these needs. There is limited rigorous evidence of the outcome of interventions aimed at changing awareness and attitudes at the individual family level. (Naudeau 2011, p.128). Nevertheless, some good practices have been identified. 36 Two types of communication campaigns are usually distinguished: (i) a downstream campaign which targets the specific populations such as families with young children, with specific messages on for example, key developmental milestones, and “talk/sing to your children”; and (ii) an upstream campaign, usually targeting a larger audience and seeking to generate public and political awareness and support for policies. It is generally agreed that communication campaigns require multiple channels and normally have comprehensive strategies to cover both upstream and downstream communications. They may use a range of communication media, depending on the technology available and characteristics of the target populations. Communication options include television (e.g. public service ads or documentaries); radio (e.g. interviews, talkback); printed material (e.g. newspapers, brochures/flyers); wall drawings, and posters; special events (fairs, plays, concerts, video shows); and information communication technology (web-based or cell-phone-based text messages) (Naudeau 2011, p 127). More recently, social media platforms such as Facebook, have been used for awareness campaigns, particularly for public health messages (Parackal, M et al. 2017). Communication programs for ECD usually target parents, grandparents, and other caregivers. However, they can also target children directly, particularly on health aspects (e.g. washing hands) through TV, cartoons and picture books, or through messaging via professionals they interact with, such as teachers and nurses. They can be undertaken by governments, or through public-private or public- community partnerships. Communication strategies are better received and have greater impact if the messages and the strategy are developed through a participatory process. Participatory processes are essential to understanding parent and community knowledge, attitudes, values and behaviors and their links to local cultural, social and religious contexts. Consultations help to identify what issues are to be addressed through the campaign (this needs to be narrow and focused), what are the barriers to people dealing with those issues, who is the target audience, what methods of delivery of messages would be most effective within the financial and time constraints available, and what agencies could be involved in the design and implementation of the campaign. In addition, involvement of the target populations in the development of the communication campaign can identify the best information channels, and the best form of the messages. In Tonga, the successful health promotion Kau Mai Tonga Netipolo (“Come on Tonga let’s play netball”) 2012-2014 aimed at women aged 15-45, was based on extensive research into current behavior and attitudes, particularly the barriers women faced in participating in physical activity. It had very careful targeting and used a range of media, ambassadors, as well as community-based communication strategies. In raising public awareness, it is important to move beyond the “what” of a message to the “why” and “how”. For example, a reading campaign could be more effective if it says, “Children who are told stories or read to, sung to and who hear and learn rhymes often have better language skills, are more confident as well as playful” (the “why”), rather than simply with messages that say “It is important to read to and sing to your child” says “what” to do. In Tonga, following extensive consultations with multiple stakeholders, it was decided to conduct a range of communication activities across the country to increase the awareness of parents, 37 communities, and Tongan political and civil leaders around the school readiness of Tongan children, in particular: (i) current strengths and weaknesses in school readiness; (ii) the importance of home-based activities and preschool in improving school readiness of children; and (iii) activities that can be undertaken by caregivers with the support of communities and churches in improving school readiness, particularly literacy. The Early Childhood Development and Education (ECDE) Advisory Council in Tonga was the primary source of advice and commentary on key messages for the campaign where all concerned stakeholders (government and non-government agencies) were involved. There were also targeted consultations with different groups of young mothers, fathers and grandmothers. Given the requirements identified in ensuring low cost interventions that don’t strain capacity of communities and government, and be capable of replication in other Pacific countries, a “light” and focused communication strategy was undertaken. From here, the “Read with your child” program was born, primarily delivered through TV and radio, plus Facebook. The decision was made based on early discussions with the ECDE Advisory Council and the focus groups to identify: (i) What were the barriers to reading – lack of books/materials? Parents not understanding its importance? It was not seen as part of Tongan traditional culture?; (ii) who should any messages be directed to – young mothers? Grandparents? Men?; and (iii) what is the best way to reach mothers/fathers? Advice and insights provided by these consultations included: i. Choice of media – ensure it fits the purpose, for example: a. As Tonga did not have a strong print/written culture, messages were better disseminated using oral or visual mediums. b. Many communities on the outer islands did not have TV, so radio had to be a key mechanism, with talkback being very popular, e.g. with women working at weaving factories/houses. c. Facebook was very popular and well used by young mothers. SMSs would be useful for connecting with younger people. d. Distinctive music and/or a jingle would be necessary. ii. Messages: need to be (a) simple, and tightly focused on something that parents can “do” in a fairly simple way, with a positive, ‘catchy’ title and simple tag line; (b) make the activity “fit busy lives” and (c) show that the activity aligned with Tongan culture, the importance of family bonds and spiritual life. iii. Role of and men, church and community leaders is critical. Without the strong and visible support of men and community leaders – e.g. government ministers, church leaders, district and town officers – no campaign will be successful. So, the messages needed to be directed to men, not to women. The campaign had to actually show men reading with children. iv. Role of the royal family or famous celebrities. 38 Finding the right delivery partner Identifying a local delivery partner to coordinate the campaign, develop content (TV, radio, social media), and to identify presenters and interviewees has many advantages. Local individuals and companies usually have good contacts and personal networks to draw on and can bring their own insights and knowledge to enliven content and presentation. However, due to the small, disbursed populations of the Pacific with lesser access to the multimedia communication channels available in more developed countries with larger population densities, many PICs have a very limited number of individuals and companies with the capacity and capability to develop the high-quality materials required for an effective media campaign. In the case of the “Read with Your Child” program in Tonga, PEARL was intent on employing the services of a local delivery partner; however, the process of identifying and contracting the right partner took more than 18 months. The long time taken to identify and secure the successful firm impacted on the duration of the campaign, and only one, 8-week campaign was possible in 2017 instead of expected annual campaigns over 2-3 years. Nevertheless, the advantages of hiring a local delivery partner in the end made up for the delays in setting up the campaign. Be clear about the objective – what exactly are you trying to achieve? A clear objective of the campaign is essential, otherwise activities can catch people’s attention but have little impact. In Tonga, after extensive discussion with experts and locals, the team developed the objective “to increase the number of children aged 0-5 being read with at home on a regular basis”. In particular, the activity being promoted was “shared” reading - not caregivers reading “to” children but reading “with” children -with both caregiver or older sibling looking at the book/words/story together with the young child. It was underpinned by the message that what caregivers do in the home makes a difference to the development of their children and how well they do at school. Developing content – the importance of context While there are many examples of public campaigns in other countries that encourage parents to read with their children, it is unlikely that this material is going to be directly usable in a Pacific Island context. In Tonga, all the campaign materials were developed locally for the local context. Features of the content from the campaign are described in Annex 3. Outcomes Measuring outcomes is essential to guide later phases and follow up work and assessing if the campaign is successful in achieving its objectives. By identifying whether the target audience received the messages, what part(s) of the campaign was/were memorable, and what (if anything) affected behaviors of the target audience (as well as unanticipated effects and audiences) provides valuable information to understand how effective the campaign was and what can be done next. Some ways to achieve this are through follow up surveys (phone, SMS, social media, written) and focus groups. PEARL was in a fortunate position to be able to “piggy back” onto existing surveys of parents and communities, which saved money and time. Focus groups were also used, feedback from ministry staff, Facebook data, and the endline census survey on school readiness using TeHCI included the question if parents had read to their child in the previous three days. 39 Overall the results were very strong on recognition of the campaign, but not on changing behavior. Informal reports from the ECDE Advisory Council, public officials and CPBAs were that the campaign was extremely visible – particularly the jingle - not only around the messages but also for raising the profile of the PEARL project in Tonga. However, the insights from the focus groups were that they could not see that it had changed behavior yet, and that there needed to be a longer campaign, with more messages targeted at community leaders. This lack of behavioral change was validated by the TeHCI 2017 survey that showed that the proportion of children being read to by parents was unchanged across the population. System/Policy-level interventions: Supporting partnerships in Early Childhood Education ECD is a multisectoral effort, requiring collaboration between multiple government ministries and non-government stakeholders, which often do not exist. Most ministries have their own mandates for different aspects of this stage of a child’s life, with Ministry of Health taking on maternal and child health, Ministry of Education taking on early education such as pre-school (if they do have a mandate below children’s entrance to primary school), and other ministries may have other programs, such as social welfare, disability and child protection. However, it’s often the case that there is no cross-sectoral collaboration on such programs, leading to gaps in service provision and lack of information sharing. The importance of multisectoral partnerships for ECD It is generally recognized that governments have an essential role within early childhood development services, particularly in health, education, childcare and child protection. In preschool or early childhood education for example, even if they do not provide preschool themselves, governments in most countries are responsible for maintaining a regulatory environment that supports quality provision and protects children from harm. They are also frequently responsible for providing financial support for parents or services, especially for the most disadvantaged children. Because in many countries ECD services are frequently provided and funded by the non- government sector, successful ECD outcomes for children can only be achieved if there are successful partnerships between governments and a wide range of non-government actors, including parents (as first teachers), communities, and non-government providers of services. Extensive partnerships are also required within governments - particularly between the education and health agencies, and the ministries responsible for childcare, families and communities (Naudeau et al. 2011). However, such partnerships are challenging for most countries. The importance of working in partnership across government and with non-government bodies is understood, but it remains difficult for most countries, not just between government and non-government bodies, but within government as well. Governments often struggle maintaining relationships with stakeholders as political imperatives and the need to maintain control of an agenda can cut across relationships. Stakeholders in turn can be suspicious and/or unrealistic in their expectations of government and government officials. Partnerships can also be undermined by weaknesses in the capacity of either side, and the failure to recognize how to deal with the power imbalance (power is usually with the government) (e.g. Findlay 2011). As a result, policies and programs that require cooperation between government agencies have a long record of failure. This issue becomes even more serious as many PICs have very limited allocations to this subsector. 40 International evidence is very clear on the key principles of successful partnership for governments. Successful partnerships deliver results. They are characterized by reaching a shared understanding of what the partnership is trying to achieve, clear roles and responsibilities of each partner, and having a way of solving problems as they arise. For successful partnerships, the key principles for governments usually revolve around: • Expectations - ensuring these are clear and understood. • Behaviors – officials are always respectful. • Building trust – officials always listen, are always clear on what they can and cannot do. • Delivering on commitments – always. This includes ensuring governments only commit to actions that are practical and feasible. • Capacity – ensuring capacity is sufficient on both sides including secretariat support for working groups/task forces or formal consultative bodies. • Solving problems – having processes and approaches to deal with problems as they arise in the relationship. Most of these are straightforward, but clarity of expectations is less well understood. A mismatch of expectations can undermine trust, respect and effectiveness of partnerships. There is a recognized continuum of how governments can engage with stakeholders/partners ranging from simply providing information, to listening to points of view of partners, to joint decision-making and to delegated decision- making. Many governments and officials, when they talk of partnership or stakeholder engagement, simply mean sharing information, possibly on short notice. Many non-government bodies will expect not only to be listened to regularly, but that they will have input into decision-making and be told of the outcome. To avoid this mismatch of expectations it is essential for governments to be clear on how they will engage, on what they will engage on, and what they will do as a result. The international literature is also clear on the importance of leadership by the government or lead government agency. The OECD has identified that the first key element of a successful early childhood policy is having a lead ministry that works in cooperation with other departments and sectors (OECD 2001, p.11; OECD 2006, p.49). Without commitment – to regular meetings, respectful behaviors, clarifying expectations - partnership mechanisms both across government and between government and non- government bodies and individuals will not be effective, and unlikely to be sustained. The experience of PEARL in Tonga shows this very clearly, as discussed earlier. Oversight and support mechanisms for playgroups When it comes to oversight and support for playgroups, this can be led by ministries of Education, Health, Community Development or others depending on the existing policies and programs in place in the country, and how each ministry identifies its own role in contributing to efforts. However, reflecting the fact that playgroups operate in a policy area where there are often no clear ministry responsibilities, it is usually necessary to have strong engagement with other ministries. 41 Ministry of Education can be a good home for responsibility of playgroups due to the strong focus of activities on cognitive development (exposure to pre-literacy, pre-numeracy, creative thinking, problem solving etc.), particularly as countries in the Pacific are increasingly extending the mandate of education ministries to younger age groups (prior to entry to primary school). Further, the play-based methods employed in playgroups are not unlike those used in preschool curricula, albeit with less formality, and therefore provide an opportunity to capitalize on existing human and other resources that lie in ministries of education. One limitation of education ministries though is that their mandate commonly begins with children aged 4 (or possibly 3) years and upwards, so they may need a formal adjustment to the age group they support. Ministry of Health generally has a mandate for programs and services for children from conception upwards, as well as prenatal maternal health, and a strong focus on the nutrition, and physical and mental health aspects of early childhood development. As such, they often have strong connections with communities and families through local health professionals. Working with the Ministry of Health can therefore have benefits in delivering important messages about the importance of playgroups and their positive impacts on families and child development to communities and families. A Ministry of Health can also provide training and messaging on health, such as nutrition, sanitation and immunizations. Ministry of Community Development/Internal Affairs are similar in many countries in the Pacific and play an important role in connecting government with communities through their responsibility and relationship with community leaders. In Tonga the key mechanism for ensuring engagement of key stakeholders in the PEARL interventions were the ECDE Advisory Council. This Council was created based on the pre-existing Early Childhood Education Council and the PEARL School Readiness Council (set up to provide initial advice on the PEARL program). See Box 2.2 for details. Unfortunately, the Council has not been sustained as it should due to political and personnel changes within MET and lack of commitment to improving cross sectoral collaboration and consultation on ECD matters. In the end, ministry officials saw partnership and consultation with non-government as more of a “problem”, and not a source of assistance and help in their development and implementation of good policy. 42 Box 2.2 - Support for the Tonga Early Childhood Development and Education Advisory Council. As part of Tonga’s efforts to improve ECE, the Education Act 2013 included provisions for ECE including registration of preschools, quality assurance, national coordinating bodies and “cross- sectoral partnerships with relevant ministries and non-government organisations working the early childhood education sector” (Sections 105-110 of Education Bill). The MET was designated as the lead policy agency in ECE, and the partnerships of cross-ministry collaboration and with the non-government sectors for quality ECE were elaborated on in the draft National ECE Policy framework and strategy, including detail on the operation of a National ECE Council. A new National ECE Council was established as the main partnership mechanism, with the first meeting held in December 2013. However, the MET had little experience of working in partnership with the non-government providers in ECE in a systematic manner. The role of the ECE Council and membership was unclear, roles and reporting lines were not defined, and no chairperson was officially appointed. People who attended the first meeting were interviewed at the beginning of PEARL - all said they had not received a membership list and they weren’t sure if all the people present at the first meeting were members or not. Subsequent meetings appeared to be ad hoc, with short notice of meetings, no forward work program, and no documented procedures. In addition, the MET had very limited expertise in ECE as this subsector was not the MET’s mandate. It was agreed that the PEARL program would support the operation of the ECDE Advisory Council, specifically, the development of the Council’s operational procedures, and to assist MET engagement with its partners, including other government agencies and non-government preschool providers. The support consisted of several actions, such as: (i) identification of risks and success factors. Discussions were held with ECDE Advisory Council members on what they considered were the important elements to ensure its success, and what were the risks that needed to be managed; (ii) technical support with the formation of ECDE Advisory Council in drafting materials to obtain formal approval from the Minister for the setting up the Council and making sure all concerned partners’ participation (government at all sectors and levels, non-government, social associations, etc.); (iii) drafting of guidelines and protocols for Council operations, including roles and responsibilities, expected meeting frequency, meeting arrangements, meeting papers, meeting outcomes and the role of the secretariat and chair; (iv) providing advice and support for Council meetings on all related operational matters, including preparation for the meeting, agenda, presentations for meetings, and recording minutes, until such time as local capacity developed to undertake these roles. The operational procedures developed included objectives and roles and responsibilities. They were short, practical documents that encapsulated the philosophy and behaviors expected of the Council and its members. It was clear that it was to act as a reference committee, providing advice and support to operationalizing ECE policy and strategy as well as the PEARL school readiness interventions. Importantly it was expected to use evidence to inform its advice. The operational guidelines of the ECDE Advisory Council are at Annex 4. 43 In Tonga and Tuvalu, it was the ministries of education that took primary responsibility for playgroups. In Tonga, with technical assistance from the World Bank to provide information and ideas, and to strengthen capacity and provide local coordination, a team of MET officers and consultants took on the monitoring role, with school principals and teachers providing leadership and support for CPBAs. During the initial stages of the intervention when there was more work to be done, such as engagement with communities, training of facilitators, and establishment of monitoring arrangements, activities were largely handled by a specially recruited team and not by existing ministry staff, making for a manageable workload for Ministry staff. Roles were also assigned to other stakeholders to capitalize on their different strengths and capacities. District nurses in Tonga with their strong connections with families and communities were key to promoting the importance of CPBAs. District officers and town officers (elected officials) play very important roles in the communities, and the Ministry of Internal Affairs provided a conduit between the lead agency and the communities through these local officials. Without their support, establishment and operation of CPBAs would not have been possible. During the later stages of pilot implementation, roles were integrated more into existing roles. For example, on the outer islands of Vava’u and Ha’apai, the monitoring was carried out by local education officers. In Tuvalu, where playgroups were integrated into the preschools (hence the intervention name playgroups@preschool), MEYS was a natural place for oversight to be housed through the local education officer in charge of ECE. Other key stakeholders that can play important roles in establishment and implementation of playgroups include churches, preschools, private sector, and community members themselves, including families. All these actors can play roles in supporting playgroups through financing, provision of venue and resources, staffing, monitoring, and promoting their importance. Lessons Learned It is obvious that (i) coordination between government- non-government partnerships and even cross government collaborative arrangements are challenging; (ii) failed partnership arrangements may be worse than no collaboration at all. In Tonga, while the non-government representatives had low expectations of the ECE Council (and the 2016 ECDEA Council) in fact the lack of MET leadership and commitment further eroded their trust in the government and in MET; and (iii) the need to make trust in government and long-term sustainability of partnership mechanisms front and center in partnership arrangements. This could be achieved by (i) having the objectives of the partnerships explicitly include sustainability, (ii) ensuring that meeting agendas always include an item on current levels of support and risks, (iii) having actions that might build commitment of government and the lead ministry, and (iv) including annual review on commitment and support. C. The value in measuring school readiness and disseminating results For countries to gain an understanding of whether their children are starting school prepared to continue their learning, children’s early development and school readiness needs to be measured. With these data come a range of opportunities (see Brinkman and Stanley 2012) that help work toward the ultimate goal of helping to support all children to reach their developmental potential. For example, such 44 data provide countries with a base level of information regarding their children’s early development, which can be used to highlight inequality across the population (e.g. is a sub-population suffering from poorer outcomes?; does development differ according to geography?; is one particular aspect of children’s development suffering more than others?), informing policy around where further investments and supports should be targeted. Such data also allow for exploration of the factors that promote child development across different contexts (e.g. what form of early childhood education is most beneficial for children’s school readiness?; does it benefit the development of all children, or some more than others?) as well as identifying what works for whom. This data can help to inform how services and supports should be targeted so that these positive influences are maximised across populations. Further, child development data can also be used to track changes in children’s outcomes over time and evaluate if policies and interventions are achieving their goals. Choosing the right instrument Tools measuring children’s early development can be broadly categorized into instruments that directly assess children’s development, and those that collect information about how children are developing via an adult respondent, most often the child’s caregiver or early years teacher. Each method has a series of advantages and disadvantages that are also important to take into account when selecting an instrument (see Fernald et al. 2017). Generally, direct assessments are conducted by trained professionals, and children are assessed one-on-one through solving problems or answering questions. Although research shows that this method can help to reduce respondent bias, directly assessing young children in this way has its difficulties, and often requires greater resources in order to train assessors and implement the instrument to each child. Instruments that rely on adult report on the other hand, are often less burdensome to conduct as they require less training and thus time and resources to administer and have been shown to be related to results collected via direct assessment which indicates that both methods are measuring children’s development in similar ways. Some research does show however, evidence of potential respondent bias (e.g. responding in a socially desirable way or responding inaccurately due to a lack of knowledge of children’s true abilities) among these instruments. Again, the purpose of measurement will likely influence which measurement route is taken. For example, if seeking to assess the impact of an early years intervention that aims to improve children’s reading ability, the best way to measure such impacts will likely be through directly assessing children’s reading skills. If, on the other hand, the measurement purpose is to understand children’s holistic development across the population, an adult-report measure will be more appropriate due to time, financial, and/or capacity constraints. This then highlights another key consideration in the selection of an appropriate measurement tool – implementation capacity. Are the funds required to cover the costs associated with utilizing the instrument available (see section on measurement costings below)? Is there adequate time to collect data via the instrument of choice (i.e. a direct assessment is far more time- intensive than an adult-reported measure)? Are people with the skills required to be able to administer the instrument available, and if so, are there enough? These are all important questions to be considered when determining if the capacity required to conduct measurement using any particular instrument is present and/or available. Another important consideration in the selection of a measurement tool is the applicability and/or 45 appropriateness of the instrument to the cultural context of interest. This is because children’s development is influenced by culture, language, and theory, and thus what are considered important aspects of child development, as well as what are thought to be suitable assessment techniques to capture this information, can vary across cultures and contexts (Hambleton, Merenda and Speilberger 2005). Consequently, it is important that a measurement tool captures aspects of children’s development that are aligned with culture as well as early years priorities, so that they not only accurately reflect children’s skills, but also produce information that is relevant to local policy and practice (Raikes et al. 2017). If an instrument has not previously been used in a particular cultural context, its applicability to that context is not yet known. If a tool is selected to be the best fit for a measurement purpose, it is recommended that local stakeholders and child development experts take part in a process that seeks to explore how the tool fits locally, and work to adapt it to be appropriate to the local context if necessary. This instrument adaptation process is described further in sections below. Finally, it is also important to consider a tool’s psychometric properties; its reliability and validity in the contexts in which the tool will be used (e.g. in a particular country or language, or with children of a particular age range or level of ability). Reliability refers to the extent to which a tool provides consistent results across respondents and/or children over time, while validity refers to the extent to which a tool captures what it is intended to measure and thus is an accurate representation of children’s true ability. Ideally, tools that have demonstrated reliability and validity over time in similar contexts will be selected for use, however it is important to recognise that this is often an unlikely scenario in low- and middle- income countries in particular, whereby measurement endeavours are often the first of their kind, and thus instruments have not been previously validated in the context. Again, for this reason, adapting an instrument to the local context is a crucial part of the measurement process, which is discussed further in the next section. For comprehensive information on the tools available to measure children’s early development in low- and middle-income countries, see “A Toolkit for Measuring Early Childhood Development in Low and Middle Income Countries” by Fernald et al. (2017). In the case of the PEARL Program, the process of choosing a tool to measure children’s development is described below, illustrating how the key points discussed above were taken into consideration (see Brinkman and Binh Vu 2017)9. a. Setting measurement purpose: the goal was to assess the school readiness of children aged 3-5 years for two key purposes: (i) to evaluate the impact of CPBAs that were implemented as part of the PEARL Program and (ii) to provide the country with reliable and comprehensive data on the state of children’s development across the country. Measurement purpose helped to define what type of measure was required. It needed to: • Measure school readiness (i.e. physical, cognitive and non-cognitive development); • Have the ability to detect changes in children’s school readiness over time (i.e. capture enough variation in children’s development to demonstrate growth along a 9 The report can be directly downloaded to PDF: https://openknowledge.worldbank.org/bitstream/handle/10986/25674/9781464809996.pdf 46 developmental continuum, rather than a deficit-based instrument that captures developmental delay); and • Be feasible for large-scale use in order to collect data from every 3-5 year old across the country (i.e. simple and quick to administer, and low cost). Note that costs are minimized when a system approach to monitoring is subsumed within the operational business of government. For example, utilizing existing staffing, such as local child health nurses and preschool teachers to collect the eHCI, and integrating it into their own professional development and evidence-based practice allows for a system wide approach to continuous professional improvement, empowerment of staff at the cold face and community engagement, while at the same time, reduces cost associated with bringing in private survey management firms. b. Context appropriate: the ideal tool would also be relevant to the Tongan context, that is, it would capture the aspects of children’s early development and readiness for school that are deemed to be important to both local culture and policy. c. Reliable and valid: the ideal tool would also have demonstrated reliability and validity in the measurement of children’s school readiness in similar contexts (i.e. for children of the same age, in low- and middle-income countries). Because children’s school readiness had not previously been assessed in Tonga, or indeed in any islands across the Pacific, there did not exist a tool that met all of these measurement ideals. The existing instrument that was deemed most appropriate of available tools was the Early Development Instrument (EDI). Created in Canada, the EDI was developed to be a population measure of children’s school readiness which has been used in over 20 countries to date. The tool can be completed by children’s teachers relatively quickly and has demonstrated reliability and validity in many high-income countries (Janus, Brinkman and Duku 2011). Despite the measurement ideals of the EDI, it requires licensing fees to be paid, and is not intended as a measure for use across diverse cultures. For example, the EDI item regarding children coming to school dressed appropriately was intended to measure children’s level of organization, rather than indicate poverty among a family who may not be able to afford “appropriate” clothing. Similarly, the EDI item relating to children stopping a quarrel or dispute is included in the instrument as this is considered a positive behavior in Western countries. In PICs, however, and in many other cultures also, this would be considered a negative behavior. Considering these limitations in terms of cost and applicability to context, the EDI was not deemed appropriate for use in Tonga, and so it was decided that a new measure of school readiness – the eHCI – would be developed specifically for this purpose. Below we summarise the key steps taken throughout this development process. For full details, see Brinkman and Binh Vu (2017). Multiple consultations with local stakeholders were key in developing an instrument that incorporated Tongan culture and priorities for early child development, as well as its relevance to local policy and practice. A series of consultations with members of health and education government departments, church leaders, as well as other locally identified stakeholders working with young children in the country sought to first understand what good child development would look like at school entry in 47 the Tongan context as well as the best method for assessing children’s development across the country. This consultation process also sought to promote local ownership of the instrument and its data. Based on these consultations, it was decided that an adult report of children’s development would be the most feasible method to collect data on all children aged 3-5 years across Tonga. Once a draft instrument was developed, on the basis of local expertise as well as the international literature around measuring development in the early years, the proposed items were translated into local language. Further consultations sought to ensure the items were capturing the main aspects of child development that were important for Tongans and that the translation was capturing the true intent of each item. Next, the eHCI was piloted among a small but representative sample of respondents in the community to ensure the items captured the full range of development among children aged 3-5 years in Tonga, that respondents understood the questions being asked of them, and were able to respond to each item as intended. Results from the pilot indicated that a mixed-methods approach was the most efficient and reliable data collection method. Learnings from a pilot should be used to further revise the instrument if necessary. For example, removing or altering items that proved to be too easy or too difficult for children’s level of ability (i.e. floor and ceiling effects), or items that were proven too difficult for respondents to provide an accurate answer to indicated by high levels of missing data. It is important to note that this process of revising the instrument and then pilot testing the new iteration may take multiple attempts in order to result in a finalised instrument that will produce locally relevant and useful results. As such, adequate time should be allocated to this process. Lastly, once the final version of the instrument has been determined, data collection protocols and manuals can be developed to help guide implementation of the full eHCI census. It should be noted that unlike student assessment tests, equating the eHCI over time is not necessary. The eHCI is a survey checklist answered by the primary caregiver and/or the child’s teacher. As such, the same instrument can be used over time and progression up or down the scales measured to determine change in child development over time. Based on the results of the pilot in Tonga, a data collection protocol was developed whereby teachers would complete the eHCI for children attending an early education program, while the instrument would be completed by parents in the community for those not attending, either via community meeting or in their homes. Finally, pilot results were presented back to stakeholders who were involved in the instrument’s development process, and the eHCI was further revised into a third and final version based on learning from the pilot (e.g. removing items that proved to be too easy or too difficult for children’s level of ability, or too difficult for respondents to provide an accurate answer to). The Tongan version of the eHCI10 (see Annex 5) measures children’s development across nine domains: physical health, verbal communication, cultural identity and spirituality, social and emotional skills, perseverance, approaches to learning, literacy (reading), literacy (writing), and numeracy and concepts. Items are dichotomous with yes/no, or can already/can’t yet response options, and are used to create nine scale scores. The number of items within each scale ranges from 4 and 13, with an average of 7 items per scale. Most items are positively worded so that the “yes/can already” response is scored as 1, and the “no/can’t yet” response is scored as 0. In addition to the items measuring children’s development, 10 Since its implementation in Tonga, the eHCI has been adapted and used in a number of low- and middle-income countries both across the Pacific as well as in other areas of the world. The eHCI was developed in the interests of public good and is available for anyone to use free of charge. 48 the eHCI captures children’s basic background characteristics, their primary caregiver’s educational level, as well as information regarding early childhood education attendance and the home environment. In summary, choosing the right instrument is crucial because, ultimately, it will play a key role in determining the accuracy of conclusions that are drawn about children’s development as a result of the measurement process. The key considerations in choosing the right instrument include: (1) whether the tool fits the purpose or goals of the measurement endeavour (i.e. does it obtain the information required to answer the research question of interest); (2) if the tools fits within any implementation constraints including those related to funding, time, resources, and capacity; (3) if the instrument is appropriate to the contextual and cultural environment of measurement (i.e. does it capture the aspects of children’s development that are deemed to be important to both local culture and policy?); and (4) whether the tool has been demonstrated to be valid and reliable in similar contexts (e.g. in low and middle income countries). The below section aims to provide a brief guide to implementing the eHCI for any country or group who may be interested. Key considerations are discussed in working to implement the tool and provide examples from implementation of the eHCI throughout the Pacific to demonstrate these steps. Census versus sample? In the PICs to date, measures of children’s school readiness through a modified eHCI tool have tended to be through a full census rather than through a sample of children. The reasons are explained in detail in Brinkman and Binh Vu (2017), which draw from the World Bank report by Young (2007). In brief, a census enables community-level data to be collected and analyzed, allowing for comparison between communities on each of the domains of child development measured by the tool. Comparing how communities do, leads to the question “why?” Why are some communities doing better than others? What are the strengths and weaknesses that help support families, children and schools in some communities better than others? And of particular interest, what are the characteristics that explain why some high-risk communities are doing unexpectedly well? These questions can only be investigated with population-wide data. A full population measure also enables identification of how children from any special sub- populations, or specific populations defined by geography, language background, or economic circumstances, are faring. And finally, a census enables robust data to be provided back to each community, so they can see how their children are faring and to raise their awareness of any child development issues. In some cases, sample data collection may take place depending on purpose of survey and other factors meaning time, capacity, budget, etc. Government counterparts and ministries Identifying government counterparts who will be engaged to help facilitate the data collection will play a large role in the success of the measurement endeavour. Many sectors have a role to play in children’s early development, and so support from a range of government ministries (i.e. not only education) is required. The team should explore opportunities to connect with health, education, finance, and statistics and planning ministries. Health and education personnel for instance, can provide support 49 around instrument adaptation, data collection protocols (i.e. who will act as respondents to the eHCI and how this will be facilitated?) as well as assisting in the data collection and dissemination processes. Statistics and planning members, on the other hand, may be able to help in identifying children to be involved in data collection (i.e. in which communities do children aged 3-5 live, how many, etc), data collection logistics, as well as data analysis and dissemination. Importantly, communicating to counterparts about why assessing children’s early development is essential, and how it will improve the early outcomes of their children, and thus the future of the country as a whole, is key to building ownership and capacity, and thus successful engagement related to the eHCI and its results. Data collection Identifying children who will be the focus of data collection will again depend on the purpose of the measurement. If the eHCI is being used to evaluate the impact of an early years’ program for example, then only a sample of the population – both those who have participated in the intervention as well as those who have not (i.e. controls) need to be involved in data collection. If, on the other hand, the purpose of measurement is to assess children’s development at a population level to gain an understanding of inequality in early outcomes, then data collection will seek to include all children aged 3-5 (and perhaps children that fall outside of this age range – again dependent on measurement purpose) across the country. Determining strategies to identify and collect data from all children in this way requires an understanding of both (i) how many children there are and where and (ii) the health and education systems in the country in order to work out the most effective way to reach the maximum number of children. For instance, there may be a pre-existing monitoring system to which the eHCI can be incorporated (e.g. a national education management information system). If preschool enrolment rates are high, this may be the most effective way through which to reach the majority of the country’s children. If immunization rates are high and children attend some form of health service or centre to receive their vaccinations at a particular time, then this could also be an effective method in which to make contact with children and their caregivers. To date, country wide censuses utilizing the eHCI have been implemented in a number of countries across the Pacific as part of the PEARL program including Tonga, Tuvalu, Samoa, and Kiribati. In all cases, data were collected using a combination of the above; early childhood teachers completed the eHCI for children attending preschool, while primary caregivers completed the instrument for their child for those not attending via community meetings as well as visits to families’ homes. Data analysis Once eHCI data have been collected, the data need to be derived into development scores for each child to enable data analysis. The eHCI scoring manual for Tonga (see Annex 6) provides an example of how development scores are derived. Children receive a score for each domain ranging from 0 to 1, and the average of these nine domain scores is taken to represent an overall development score, also ranging from 0 to 1; with a higher score signifying better development. Data analysis can then explore a number of research questions that are dependent upon the purpose of the measurement. For instance, children’s development can be explored across gender, age, geographical location, socioeconomic status 50 (i.e. household wealth or caregiver’s level of education) as well as developmental domains. Analyses can also work to explore key predictors of children’s development such as stimulation in the home environment and early childhood education participation. Moreover, with multiple waves of eHCI data, changes in children’s development over time, and possible influences, can be explored. Data analyses can also help to determine if the instrument is working well in a new context by ensuring certain expected results and trends are observed. For example, analyses would show if development scores discriminate across children by age, gender, and socioeconomic status, as would be expected, like older children receiving higher scores than younger children, children from more advantaged backgrounds receiving higher scores relative to their more disadvantaged peers, etc. The importance of disseminating results Another critical step in the process of measuring children’s early development is the sharing of results. Although dissemination often receives little attention relative to data collection itself, it is only through the process of sharing learnings that awareness of children’s outcomes and the drive to improve these can be promoted. Although dissemination of results can often refer to academic dissemination through the publication of written material or presentations at conferences, here it is the importance of local-level sharing of learnings that is the focus. Effective data dissemination will take place at multiple levels and through multiple avenues. Data should be provided to key stakeholders, policy makers and service providers in the early years space, as well as those in related fields such as health, planning and finance. Beyond simply providing results, efforts should be made to engage these key stakeholders with their data, helping them to understand what results mean and how the data can be used to guide their work. Results should also be communicated back to any stakeholders who were involved in both the instrument development and/or adaptation process, as well as data collection processes in order to build on local ownership of the instrument and its results. Finally, and crucially, results should also be provided back to communities. Presentation of community results, in combination with some brief information relating to the importance of children’s early development and what helps to promote positive developmental outcomes, can be powerful in building local ownership, capacity and community-wide interest in both results and the mission to improve outcomes for children. Dissemination of results is recommended to be a key final step in the process of measuring children’s early development and school readiness, and a comprehensive dissemination plan involving a range of stakeholders can help to maximize both the reach and impact of results. Below, we summarize data dissemination processes that took place after the national census of children’s school readiness in Tonga to highlight this process in action. In Tonga, results from the eHCI were collected from all communities across the country, so data dissemination strategies sought to share results in the same way. Results were provided to stakeholders and communities through various avenues, which are briefly summarise below (for full details, see Brinkman and Binh Vu 2017). The official launch of the data included the Tonga’s princess, church leaders, and ministerial and development partner representatives. The event was later broadcast on local television and radio networks. Next, community meetings were organized to present data to each community. Training around the communication of results as well as the basics of children’s development 51 and the importance of early child stimulation to promote healthy development was provided to health and education personnel, whom then went back to their communities to share results at town meetings. Finally, stakeholders and policy makers, including members of the Statistics Department, Ministry of Health, Ministry of Finance, early childhood education service providers, donor organizations, and church leaders were supported around the understanding and use of their data. Presenting results visually, through geographical mapping, proved to be a successful results’ sharing strategy. Data were grouped according to village and then mapped by the Department of Survey and Lands. Local consultation determined that the “traffic light” system would be used to indicate how well children were developing across communities, with each community provided a color on the basis of their eHCI results. Aggregated community results were ranked from lowest to highest, and the bottom third of communities were colored red, the middle third colored orange, and the top third best performing villages colored green. The resulting maps provided child development results in a visual format that are easily understandable (see Figure 2.4). Importantly, maps helped to prompt conversations around the importance of early childhood development, how children are developing in one community relative to another, and what was working well in communities who were demonstrating good child development. Figure 2.4: TeHCI 2014 Mapping - Overall Development for Tongatapu Source: Brinkman and Vu (2017) In some countries, government departments will have personnel with expertise in mapping and/or geography, and in some cases, country maps with communities defined will already exist. In the case of Tonga, previous donor funding was provided to the Department of Survey and Lands for the creation of tsunami evacuation plans. As such, the software needed for geographical mapping, as well as the capacity and skills required, were pre-existing and were able to be leveraged for the purposes of mapping eHCI results. This enabled community-level results from the eHCI to be geographically mapped to communities that were locally meaningful, having already been locally defined and geocoded. If geographical mapping of results is something a team would be interested in developing but the maps 52 and/or expertise do not already exist in country, this will need to be considered in project costing (see below). Discussing TeHCI results on maps at community meeting, August 2014 (Photo: Quang Vinh Nguyen) The national eHCI data collection in Tonga acted as the first part of the evaluation of the PEARL program, with the intention to repeat the process after implementation of the program was completed. Therefore, data dissemination sought to build community ownership and capacity in the hope that the monitoring of children’s development would become embedded into the country’s systems and be as cost effective in future as possible. Finally, regarding cost and timing, although the instrument itself is free of charge, the costs and timelines associated with adapting and implementing the eHCI are an important consideration. Budget and timelines will vary depending on the country context, project purpose as well as sample size. A summary of key budgeting items for consideration is provided in chapter 4 to illustrate the costs of each aspect of implementation. Few notes to take in considerations in term of timing plan for implementation: • Project preparation: initial meetings with government counterparts to plan project (2-4 days, funds required to cover meeting space, travel). • Instrument adaptation: o Local consultation meetings (approximately 5-10 days, funds required for meeting space, travel); o Expert time for adaptation and translation (5-10 days); and o Pilot testing and analysis (5-10 days, funds for enumerator training, travel costs, materials i.e. tablets, enumerator salaries and/or per diems) • Data collection (timeline dependent on country and sample size, funds for enumerator training, travel costs, materials i.e. tablets, enumerator salaries and/or per diems). • Data analysis and preparation of results (4-6 weeks). • Data dissemination (formal launch, communications training, geographical mapping, delivering results to communities which requires travel and per diems – timelines varied). 53 D. Conclusions The importance of investing in improving early childhood outcomes and school readiness of children is well established but not always well-known. Designing and implementing interventions that ensure all children have the opportunity to reach their full potential benefits societies by increasing human capital and economic returns. Key requirements prior to scaling up interventions include: piloting based on global evidence of what works, but adapting to the local context; establishing and maintaining partnerships between government, non-government and private stakeholders as well as cross-sectoral coordination within government; monitoring implementation to resolve issues that arise in a timely manner; and measuring ECD outcomes over time to assess the effectiveness of interventions as well as understand what else is going on that is affecting children reaching their full potential. 54 Chapter 3 – Designing Interventions to Improve Early Literacy A. How to identify interventions to improve early literacy? What works to improve early literacy skills? The collective experience of the last decade has identified three main factors that underpin interventions to promote effective early grade literacy in developing countries: (i) policies and programs that support a wholistic developmental approach to early childhood development to build strong early literacy – also known as pre-reading—skills; (ii) early grade reading interventions that focus on delivering structured instruction in the first grades of primary education to provide the needed foundations for reading; and (iii) a holistic approach to interventions where an intensive intervention at one level is accompanied by complementary actions in others to create synergies across levels and treatments. The first two aspects are based on the recognition that “skills beget skills” and that there is a window of opportunity to develop early literacy skills that will lead to a lifetime of learning. The last element acknowledges that multiple environments (i.e. home, community, early learning centers) coexist during the early years and the transition into school, thus demands integrated policies that support children during these periods and transitions. In the same years, action research projects around the world have moved from diagnostic to interventions identifying reading through variations of the “five core components” of reading: (i) phonemic awareness, or the ability to identify the individual sounds in spoken words; (ii) phonics, or the correspondence of letters (graphemes) to sounds (phonemes); (iii) vocabulary; (iv) fluency, which is the ability to read text accurately and quickly, with natural prosody (tune and rhythm of speech and how these features contribute to meaning); and (v) comprehension, which is the ability to understand and communicate meaning from what is read. These five core components are basic foundational skills needed of reading ability, which are “intervenable”, i.e. improvements in outcomes can be seen with adequate practices. Key elements of early literacy Reading Reading is a complex cognitive process and we know from research that an effective reading program must address several aspects of reading. It is important to understand that none of the essential components of literacy development of reading alone is sufficient. The interconnectedness of each of the components makes it impossible to teach them in isolation. The elements needed are explained as follows: Vocabulary Vocabulary refers to the words we must know to communicate effectively; words in speaking (expressive vocabulary), and words in listening (receptive vocabulary). Children use the words they hear to make sense of the words they will eventually see in print (Neuman and Dickinson 2011). Vocabulary plays an important part in learning to read. Beginning readers must use the words they hear orally to make sense of the words they see in print. Kids who hear more words spoken at home learn more words 55 and enter school with better vocabularies. This larger vocabulary pays off exponentially as a child progresses through school. Vocabulary is key to listening and reading comprehension. Readers cannot understand what they are reading without knowing what most of the words mean. As children learn to read more advanced texts, they must learn the meaning of new words that are not part of their oral vocabulary. Language Development Long before children learn how to read words on a page, they develop and hone the skills needed to understand how language works. Rapid growth occurs in the language centers of the brain during the early childhood years. Before the age of 8, children form the foundation for language and literacy development by discovering that speech has patterns and the symbols they see in the pages of books have meaning. The course of language development is very similar across children and even across languages, suggesting a universal biological basis to this human capacity. The rate of development varies widely, however, depending both on the amount and nature of children’s language experience and on children’s capacities to make use of that experience. Research has proven that early literacy is closely linked to language development in the preschool and kindergarten years. Thus, disparities in ethnic, linguistic and socioeconomic backgrounds contribute to language skills’ delays and smaller vocabularies that cause students to fall far behind their peers. Therefore, it is critical to help children gain as much vocabulary as possible well before they start primary school, meaning from home environment and at preschool or kindergarten. Print Awareness Print awareness is a child's earliest introduction to literacy. Children with print awareness understand that print has different functions depending on the context in which it appears – e.g., menus list food choices, a book tells a story, a sign can announce a favorite restaurant or warn of danger. Print awareness is understanding that print is organized in a way - e.g. knowing that print is read from left to right and top to bottom, or in other directions for particular languages. It is knowing that words consist of individual symbols that are called “letters” and that spaces appear between blocks of letters or “words”. Sounds of Speech To understand a spoken language, a child must be able to hear and distinguish the sounds that make up the language. Most children can distinguish between different speech sounds in their native language -e.g. between grow and glow. Children who are unable to hear the difference between similar- sounding words will be confused when these words appear in the same story or sentence, and their comprehension skills will suffer dramatically. Educators and parents help children increase their initial knowledge of sounds of speech through play, figuring out words that rhyme, coming up with words that share a beginning sound, and saying silly words. All help build a child’s phonological awareness; that is, the ability to notice, think about, and play with sounds in words. Phonemic Awareness Phonemic awareness is the ability to notice, think about, and work with the individual sounds in spoken words. Manipulating the sounds in words includes blending, stretching, or otherwise changing words. Children who cannot hear and work with the phonemes of spoken words will have a difficult time 56 learning how to relate these phonemes to letters when they see them in written words. Narrative Skills Storytelling encourages children to use language in a concrete way. Whether they are recounting a tale that they just heard or sharing a personal experience, they are learning about syntax, i.e. the structure of a sentence, reinforcing comprehension and building their vocabulary. Strong readers think actively as they read. They use their experiences and knowledge of the world, vocabulary, language structure, and reading strategies to make sense of the text and know how to get the most out of it. Language Comprehension The ability to construct the meaning of spoken language, or language comprehension, requires a complex mix of different abilities. Two large domains of knowledge are required for success. The first is linguistic knowledge, or knowledge of the formal structures of a language. The second is background knowledge, or knowledge of the world, which includes the content and procedural knowledge acquired through interactions with the surrounding environment. For example, if a conversation revolves around “dining out”, a child will understand what the conversation is about if the child knows either through experience or meaningful exchanges, that “dining out” typically involves attending a restaurant, sitting at a table, being served food, eating, paying the bill, and leaving. When a child has well-developed schema in a domain of knowledge, then understanding a conversation relevant to that domain is much easier because the child already has a meaningful structure in place for interpreting the conversation. The combination of our linguistic and background knowledge allows us to make inferences from language (Wren 2009). Act on evidence - Assessing early grade reading skills Diagnostic Assessment To shift the focus of education improvement from access to achievement, it is critical to determine how serious and widespread low learning levels are among a country’s students. As a first step, measuring how well students read can make policy makers, educators, and all stakeholders more aware of reading levels and what the implications are for future learning. Such awareness can lay the foundation for discussions of how to best address problems. The EGRA11 is one tool used to measure students’ progress toward learning to read, which PEARL adapted and used in baseline surveys in several Pacific countries. It is a test that is administered orally, one student at a time. In about 15 minutes, it examines a student’s ability to perform fundamental prereading and reading skills. In the Pacific, most national and international assessments are paper-and-pencil tests administered to students in grades 4 and above; thus, they assume students can read and write. The findings of such tests cannot always tell us whether students score poorly because they lack the knowledge tested by the assessments or because they lack basic reading and comprehension skills. Teaching young children to read is the cornerstone of improving educational outcomes and has 11 Since 2006, the Research Triangle Institute (RTI) International, with the support of donors, has worked with education experts to develop, pilot, and implement EGRA in many countries and many languages. 57 far-reaching implications. Unless they learn to read at an early age, children cannot absorb more advanced skills and content that rely on reading. Children who do not learn to read in the early grades risk falling further and further behind in later grades, as they cannot absorb printed information, follow written instructions, or communicate well in writing. These challenges, rooted in poor reading skills, lead to disappointing results and often early dropouts from the education system. In the aggregate, reading and learning achievements are central to economic productivity and growth, and EGRA is a useful tool to inform the design of interventions to address the issues identified. Setting objectives and priorities for early grade reading program Typically, education curricula include the five core components of reading in the scope of learning for the early grades. Sometimes, curricula expect outcomes to happen too early or too late as a result of misalignments about when it is reasonable to set up expectations. Sometimes the curricular expectations are at the right level, but there are not clear definitions around the expected outcomes and/or their measurements. In other cases, teachers have not been trained on the methods that are most efficient to promote early grade literacy either in pre-service or in-service training. In multilingual contexts, language policy may not allow for sufficient instruction in oral language development, vocabulary and text reading between language one (L1) and language two (L2)12 or others. Additional elements that may complicate the implementation of language polices include teachers without the language skills to teach in a particular language (L2), or insufficient reading and instructional resources to help students strengthen L1 and L2. Linguistic transition practices between L1 in the early years and L1/L2 in primary schooling may be absent from the school experience. Moreover, cost considerations may result in insufficient resources to promote reading development, which are different to reading for research, reading for pleasure, and reading for understanding in upper primary grades. To set up priorities for early grade reading, it’s important to use data to help define not only what’s intended to change (definition) but how much it should change (measurement). Assessment data is one important ingredient for determining priorities. In addition, determining language of instruction policies with the understanding that mother-tongue based multilingual education (MTB-MLE) in the early grades is best instructional practice and provides a strong cultural preservation mechanism among communities., It’s important to note that in a linguistically dense country, language mapping can be particularly useful for developing a multilingual approach to early grade reading, although for the three EGR pilot countries under PEARL it was not necessary to conduct language mapping. It’s also important to identify disadvantaged groups to ensure their inclusion, determine teacher education policies related to recruitment and deployment, specifically related to mother tongue, and determine instructional time, materials for teaching reading and language of instruction as well as availability of books and textbooks in language of instruction. All these should be taken into account in defining the priorities and the 12 L1, or first language, is what is referred to the native language of the student. It is also referred to as the "natural language", or the "mother tongue". It is the language with which the student is most comfortable speaking because of the extensive exposure to it. L2, or second language, is also known as the "target" language. Any other spoken system learned after the L1, is considered an L2. The L2 can either be a "second" language or a "foreign" language. 58 objectives that could be achieved at the end of the program implementation. Discussion and policy dialogue to set up priorities Using initial results is crucial to gather political support to move from diagnosis to action. How results are communicated are also important; avoid blaming specific actors, gather support around common goals, provide clarity on what needs to be done and how these actions respond to the needs (relevance) of various actors. There needs to be a balance between directive and consultative actions. Stakeholders must gather to identify key actions that need to be included, their perception of ranked importance of the actions, and their ability to support any actions agreed. This is important to ensure stakeholders, including those from outside government, feel represented and part of the response actions, as opposed to just beneficiaries or recipients of the actions. Identify resources and ways to integrate in a period of 3-5 years (initial intervention) and 5-10 years (roll out) In most countries, national resources are limited. International development programs often compete for attention and place in government budgets and identifying local human resources to support program implementation. Programs have greater chance of success in countries where strong partnerships evolve, and have the national government leading coordination of support and assistance to maximize all resources for sector priorities and avoiding competition and duplication of efforts. This is crucial as sometimes competition among programs or projects could cause confusion to teachers, students and parents, particularly when they use different approaches or methodologies. It is critically important to identify and assign staff to work on the program right from beginning, including the work load allocation and clearly defined attributions (roles, decision-making authority, etc.) of each staff to act in their new capacity within the program (syllabus developer, teacher trainers, coaches, or enumerator, etc.). Deciding on the scope and scale of an intervention Defining the scope of the intervention This is a critical step as it involves agreement with government counterparts on level, intensity and scope of the intervention as well as the expected outcomes. Defining the scope of the intervention involves the grades, desired outcomes and key metrics of success. These need to be aligned with national priorities, but also strike a balance between ambition (having a transformative effect) and realism (how much can be achieved in a pilot in a given time frame). Setting realistic expectations are crucial for program operation (planning, execution, monitoring and evaluation), communications and engagement with different stakeholders, and resourcing. A key issue in deciding the scope of the intervention is its integration within the government structure, including considerations around resources and capacity, governance, collaboration, and institutional readiness to undertake the activity. This is important to ensure implementation succeeds and builds local institutional capacity for sustainability. Another key consideration is the desired level of rigor of program evaluation in order to inform modifications and scale up. Incorporating a randomized control trial impact evaluation model into the design provides the most robust evidence of the impact of the program, but it’s important to understand 59 and address multiple potential challenges, including budget, expertise, social perceptions around maintaining a control and treatment design, and feasibility of reducing design contamination (i.e. spillover effects of the intervention being partially or fully implemented in control schools). The program design needs to be clear on the intended outcomes and the theory of change supporting it, i.e. the different elements of the intervention, and how they interact to support and moderate intermediate and final outcomes. Expectations and requirements at each level are also crucial around operational structure, government oversight and delegation, financing arrangements, monitoring and evaluation, and communications. B. Designing and implementing multi-level interventions for early grade reading Learning to read is influenced by factors at many levels Although learning ultimately occurs at the student level, it is embedded in and influenced by factors at multiple levels. These levels range from family, school and community, to system-level factors (Bronfenbrenner 1979). Therefore, in addition to understanding child-level factors that contribute to literacy acquisition, it is imperative to address issues at family, community, and system levels to ensure that all children learn to read in the early grades (Kim et al. 2016) (see Figure 3.1). Figure 3.1: Factors contributing to literacy acquisition Source: Kim et al. 2016 The first level of influence pertains to the young child who later becomes a student. Factors such 60 as cognitive ability, sensory capacity (e.g. hearing impairments), and physical and clinical conditions (such as early language impairments) have demonstrated to have an influence over children’s ability to develop early literacy and early grade reading skills. The interaction between nature and nurture is essential for cognitive development, and while the rate at which specific cognitive skills develop may vary across cultures, all of these skills are available to support children’s learning (Fischer and Silvern 1985). Further, because of the high level of developmental plasticity exhibited by humans, i.e. changes in neural connections resulting from interactions and learning opportunities, children can develop skills when provided the opportunity (Linan-Thompson 2014a). A child’s family presents the second level of influence, with his or her early environment supporting mastery of early-age reading skills. In the nurture-nature equation, parent-child interaction is the first point of contact for children to experience learning. Research suggests that differences in the quality of the literacy environment exist between different socioeconomic groups (Anderson and Stokes 1984; Feitelson and Goldstein 1986; Mason and McCormick 1981; Miller 1969; Ninio 1980; Raz and Bryant 1990; and Wells 1985) and that these differences are ultimately related to child language ability (Mason and McCormick 1981; Wells et al. 1984). Opportunities for literacy-related activities such as reciting rhymes or looking at books at home will have considerable effects on language development and reading success of children (Payne, Whitehurst and Angell 1994). Though evidence shows that despite the economic difficulties and other stresses faced by the low-income families, many still manage to engage in interactions such as shared picture book reading that are motivated by long-term goals such as school readiness. This suggests that the home literacy environment of children provides a window of opportunity to help low-income or disadvantaged families support reading outcomes as discussed in the previous chapter. As children become students in pre-school and primary grades, parenting to support literacy skills evolves as well. Engagement with written text through adult-child reading activities have a demonstrated effect of fostering curiosity and a positive emotional connection with the act of reading which can later ignite a love for books and reading. The school and community level provide a third level of influence. When children become students in the education system, they transition from the familiar environment of home and other non- formal setting into a new environment with defined rules, goals and expectations that will provide new experiences and environments for learning. The school environment includes both the physical buildings that make up the school classrooms and playgrounds, and the policies and practices that support teaching and learning, health and safety of students, and the interaction with the broader community. It is in the school environment where children first experience three sets of factors of great significance to their ability to thrive as readers: the language context, since this is the vehicle for learning and understanding; the quality of instruction; and the partnerships and collaborations between school leadership and the community at large. The ability of school actors to organize together to deliver effective literacy opportunities in the classroom and in the school as a whole, and to foster partnerships with the community at large becomes instrumental to help children become fluent and confident readers. Yet, schools are limited in the flexibility they can exert to foster literacy-promoting environments. Finally, the education system comprises an intentional arrangement of a wide range of actors trying to guide and support student learning. In this definition, students are tasked with learning while the rest of the actors guide learning through direct instruction (teachers), organization of school resources 61 (school leadership), and general support activities (community and parents). In this environment, the decision-makers are responsible for the overall organization of education resources at a given level and as such, must define the guidelines, policies and expectations of learning. Three areas of policy and system-level action are particularly relevant to the effectiveness of the system to help children learn to read and write in the early grades: teacher policy, language policy, curriculum and resources, and financing. Components of a multi-level early grade reading intervention At the school level of influence, EGR interventions have been designed and implemented providing a set of intentional actions grouped in a program aimed at changing practices, attitudes and behaviors around teaching and learning of reading in the early grades. They include: (i) an instructional component that focuses on improving the content (skills promoted), time on task, and interaction with text, through explicit, intentional activities that promote text-based abilities and exposure to texts of various meanings and purposes; (ii) a teacher professional development component (helping teachers deliver new instructional material/activities) providing teacher support in training workshops (face-to- face and distance activities) and coaching; (iii) a monitoring and supervision component to ensure the program is delivered at the classroom level as intended, and inform adjustments as needed; (iv) a family and/or community component that links reading and reading materials both inside and outside the school; and (v) a system component in the form of policy-informing activities has been included in EGR interventions, but in limited cases. Four of these five have been implemented in Tonga’s EGR program13, “Come Let’s Read and Write” from 2015-2017, with core components also implemented in Kiribati and Tuvalu under PEARL. Together, these components form the basis of an evidence-based EGR approach. Providing an instructional guide and training on how to use it are the core elements to ensure instruction is focused on fostering predictors in an efficient (time on task) and appropriate way (modelling of the activity). Tools for student assessment complement the package linking instruction with results: how to adjust instruction and support struggling readers (monitoring/formative) and how effective the intervention is to deliver learning expectations as per the curriculum (evaluation/summative). The provision of coaches aims to support teachers improving their classroom practices during the school year to guide pedagogical improvement and reflection, and to gather information around implementation and pedagogical transformation that may be useful for future iterations of the program. Additional reading materials reinforce learning by providing targeted practice for children (decodable, short stories) and foster reading for enjoyment, which can also be emphasized in the home and community environments. Monitoring and supervision also aim to improve implementation fidelity and provide important information with which to inform changes at the program- and system-level, such as policy and resourcing adjustments. Below, these components are explained in more detail as well as the considerations on how to design, implement and monitor effectiveness of each component. 13 The “family and/or community component” was planned for Tonga in the form of a public awareness campaign, but did not take place due to delays in setting up the first campaign under the School Readiness component. 62 Instructional approach Student-level outcomes are affected by the EGR instructional approach followed. These can be defined concretely by an instructional guide for teachers, i.e. a teacher guide (TG), and supplementary learning materials for reading, e.g. decodable books, short stories. These two complementary materials are prepared to provide adequate guidelines for instruction and to allow students to practice and progressively build on the new knowledge they absorb with each lesson. A review of EGR instructional guides in challenging contexts in developing countries (Piper et al. 2018) confirmed their effectiveness to support teachers in delivering reading instruction, and that the ideal guide should provide step-by-step instructions without too many words or complex procedures. Despite the differences in the scope and modifications to the instructional guides used in EGR programs to date, many share the orientation and activities required to guide EGR instruction. The same cannot be said of supplementary reading materials, though this is not necessarily an issue of concern. Decisions over scope and skills covered in the supplementary materials are very much context-dependent, such as availability of existing materials and teacher capacity to use existing resources/activities to allow students to practice. Developing teacher guides Creating an instructional process for EGR programs requires technical expertise and acute knowledge of the language and context. It cannot be conducted only with external assistance but requires the presence and continuous support of in-country experts during the design, pre-test and roll out stages to address technical questions and disputes that may arise. While the process to develop a TG will vary according to the context, the following presents the six steps followed under PEARL. These steps ensure TG content promotes foundations of reading and provides students with a coherent sequence of content as well as assessment practices that match the corresponding content standards within the curriculum expectations in a particular grade level. Step 1 - Conduct a curriculum review of standards, curriculum expectations and resources used in the early grades. This is an essential step in the process to ensure the program aligns with official curricula. It also allows for an identification of evidence that will support the scope and materials for the EGR program, and the expertise that will be needed to deliver it (inside and outside government, at the school level). Under resources, this step should also include a review at the policy and programming level of parallel initiatives for EGR to avoid duplication, perceived competition, and confusion with teachers, parents and school leaders. The outcome of this review will produce an identification of outcomes to be improved, the expected change, the program requirements for success (design and implementation), as well as decisions around rigor and purpose of evaluation approaches. The consensus reached will be the policy and program framework that will guide EGR efforts during the period of initial intervention. Step 2 - Ensure alignment with the EGR curriculum and the national expected learning outcomes. The relevant curriculum would be reviewed to identify expectations related to foundations of reading, such as which grades, how teachers are advised work on this, integration across skills. The following questions should seek for answers: What are the pedagogies teachers know already, what is missing? How are these skills currently observed, recorded and measured in the classroom and/or in the system? In the experience of teachers, how useful is the data from these assessments to inform instruction (formative assessments) and capture learning at critical milestones of learning (summative)? 63 A structured consultation workshop was used to complete this activity. In preparation for this workshop, a focus group with pre-service and in-service teachers training providers was organized to understand the country’s current implementation processes and challenges. At the workshop, a prototype outcomes map in reference to existing curriculum expectations could be developed, discussed and revised. The defined expected learning outcomes at the end of each grade are used to lead the materials development. Step 3 - Material development. PEARL EGR programs use the Gradual Release Model to guide instruction of letter knowledge skills, phonics, word recognition and decoding. It’s a best practice instructional model where teachers strategically transfer the responsibility in the learning process from the teacher to the students (Fisher and Frey 2008). This model (illustrated in Figure 3.2) provides teachers with an instructional framework for moving from teacher knowledge to student understanding and application. This approach is widely recognized because it features four phases that model each concept: “I do”, where the teacher shows the students how it’s done; “You do”, where students practice the activity together; “We do”, where the teacher and students do the activity together; and “You do it on your own”, where the students practice independently based on the modelling of the activity. Figure 3.2: Gradual Release of Responsibility Model Source: Fisher and Frey 2008 Figure 3.3 includes key guidance for the production of new or revised EGR TGs. 64 Figure 3.3: Essential Guidance for New or Revised EGR Teacher Guides Source: Piper et al. 2018 Step 4 - Pre-test lessons in a small sample of classrooms. To ensure the materials developed are relevant and appropriate, pre-testing some lessons is crucial prior to finalizing the materials for teacher training. Questions to answer as part of this exercise include: Do the lessons make sense? Are they clear for teacher and students to follow? Are lessons too long (which could lead to instructional exhaustion)? Are the students engaged? Is there time in the reading/language lesson for independent work? How/when should this be included? What is the feasibility and main challenges to initial take-up by teachers and students. The initial three weeks in Grade 1 are key to pre-test, as this is when students are still new to the formal classroom setting, and teachers are learning a new approach to reading instruction. Based on the results of the pre-test, the lessons are revised and prepared for printing and distribution to schools. Format and layout of the materials is important but should not be the priority at this stage: it’s recommended to adapt existing guide formats already in use in a particular context to ease teacher familiarization with the process. Step 5 - Periodic TG Review. As the TGs are used by teachers in EGR classrooms, the materials are constantly put to test on their efficacy to support EGR skills, and the ease and engagement they allow during instruction. After a set period of implementation (typically every school term) data and feedback are gathered from teachers, coaches and monitors to understand what’s working well and what isn’t, allowing for further revision of materials prior to continued use or scale up. Ongoing monitoring and recording of feedback from teachers is important to inform revisions, but decisions about what to change 65 should be centralized in the technical team. Modifications by teachers during pilot are not encouraged before the pilot period ends because it is important to allow for the treatment to deliver effects. Step 6 - Final revisions, integration with curriculum, agenda for next phase. The ultimate goal of an EGR program is to kickstart an iterative process of continuous improvement in EGR instruction and practice in a given context. As a result, it is important that once the pilot stage of the EGR program is completed, the final lessons are distilled and discussed in technical workshops as well as to the wider community. Actors involved at different levels will be interested in understanding the results of the efforts they’ve made and it’s important to close the implementation loop with reflection and discussion of results - what went well, where did we achieve results, where did we miss the mark and why? The focus is to learn from the experience. The reflection and final revision stages set the agenda for the next phase in the iterative model which could be maintenance or scale up. The decisions could also signal the right moment to add other areas such as mathematics to build on the lessons gained. It could also be used to bring information to policy makers and donors to support an integrated program of support that creates synergies and complementarities across different donor-supported programs. Supplementary materials The process to create supplementary materials14 follows a similar flow to the production of the TG in terms of alignment with the curriculum. During development of supplementary materials, the TG must be closely followed to ensure students have additional reading materials that reinforce the concepts seen in the same sequence as in the TG. While mimicking the process of TGs in the sequence of production, pre-test revision, and socialization, the process requires different skill sets for writing short stories and decodable stories, producing illustrations, and developing activities. It is also crucial that a linguist be involved to provide quality assurance to content and integration with TGs. Managing story production The process for story production is very similar to the production of the TG and should build on the products, guidelines and resources reviewed for the TG. However, given the intensity of work this may require a dedicated team. In many countries, teachers are often expected to write their own stories. However, to ensure alignment with the TG and consistency across classrooms implementing the interventions under PEARL, the team decided to develop these centrally alongside the TG. This had implications for the team’s workload under the PEARL program. Identifying who will be a good candidate both in terms of expertise and talent is crucial for the quality of the stories and the scheduling of material production. Ideally, there’s a dedicated activity manager for this activity that works closely with the lead for the development of instructional materials. If not available, the lead can do this activity. There are benefits of having a single individual leading the two tasks in terms of awareness and in-depth knowledge of the material produced and the performance of experts and personnel involved in the task. However, there are also capacity constraints and the more tasks that the individual leading the development of TGs and supplementary materials is involved in, the higher the risk of delays in the execution of the production 14 The supplementary materials developed under PEARL in Tonga were: Teacher Assessment Book, Student Activity Book, Student Readers (read alouds and partner reading), Student Writing Book, Homework Book, and a Poster Book (of illustrations depicting scenes from stories). 66 schedule for both activities. Producing guidelines for supplementary materials development is crucial. Guidelines provide the scope, themes and characteristics (lengths, structure, format) of the stories to be produced. This is not only important for quality assurance but also in terms of content; striking the right balance to ensure students are challenged without content being too difficult to comprehend is critical. Effective guidelines for the development of supplementary materials provide an explanation of the task to be done and why it is important that the outputs conform to the characteristics provided. This promotes a clearer understanding of the task ahead and self-assessment of the quality of products. When teachers are tasked with creating a battery of supplementary materials, good guidelines to produce supplementary materials appear to improve ease of use, improved metacognition of the theory supporting EGR skills and ways in which they need to be used to reinforced learning. The Global Reading Network15 and its alliance partners have already produced guidelines to develop decodable and leveled books. The process includes information for English language materials but it’s adaptable to most languages. Role of Illustrators Drawing illustrations for educational materials is a skill not all artists are necessarily capable of. Illustrations are very important in children’s books as they help young readers relate with the stories more easily. In the context of instructional materials, illustrations serve a key purpose; they must convey a message or point that relates to the bigger context or construct depicted in the story. It can be said that half of the work is done once the scope of illustrations are mapped to the content material as it explicitly defines the concept and presents suggested imagery. However, the illustrator should ensure that the images produced are exciting, culturally relevant and age-appropriate for the intended audience. Illustrators should also be good at taking detailed instructions while remaining true to their creative expression. The illustrations can be made by a dedicated artist or they can rely on digital images from a bank of illustrations for digital books. Printing materials and digital technologies Time and budgetary constraints will impact on how the materials are delivered to teachers and classrooms, both in terms of format (printed vs digital) and modality (delivery methods). In many Pacific countries, printing on paper locally is substantially more expensive than contracting an international printing house with experience in publishing, and then delivering it to the country. If the printing is procured overseas, ensuring the timely delivery of books in country and from there to the classrooms is crucial and requires careful planning of supply chain logistics. In Tonga and Tuvalu, printing houses were contracted in Australia, New Zealand and Fiji to print and deliver materials, while in Kiribati, the Ministry’s in-house printing center took care of the program’s printing needs due to the short turnaround required. In terms of quantities of supplementary materials, the goal should be to have a book for each student, but if resources do not permit, the ratio should not exceed one book to two students, as it’s difficult for more children to read from the same resources effectively. Recent international experience 15 https://www.globalreadingnetwork.net/, the network supported by the USAID seeks to improve the impact, scale, and sustainability of primary grade reading programs. 67 with Bloom16 has opened the door to the digital production of books around the world. The procurement of inexpensive devices to be used as e-Readers have the potential to bring multiple materials to children in poor literacy resource environments. Progress in the international community to create a global digital library of children’s books brings the promise of a wide range of EGR-appropriate materials. Teacher training and continuous support Teacher training is a crucial next step in implementing an instructional approach that will ultimately affect student-level outcomes. Many teachers in developing countries have little-to-no training in teaching literacy and are often undereducated in general (Pryor et al. 2012). To fill this gap, effective EGR interventions provide capacity development to teachers in multiple ways. Because teachers are expected to introduce an innovation into their classrooms, training should allow for enough exposure to the intervention (“what am I supposed to do?”) as well as continuous support to ensure teaching practices improves. First of all, teachers are trained typically through a series of intensive trainings. Second, continuous support is provided to teachers primarily in the form of coaching (see section below). Teacher training Teacher training on instructional guide: EGR interventions often train teachers using literacy curricula that are appropriate to the context and based on sound evidence for how to teach literacy to a specific group. EGR interventions have used different modalities to train teachers in the use of the EGR TG. Given the importance of modeled practice and peer-feedback, the face-to-face model is favored above others. This is typically delivered in the form of workshops or conferences facilitated by EGR lead implementors or training of teachers. Under PEARL, teacher training workshops were conducted in central locations, either in country capitals, or regional centers (such as Tongatapu, Ha’apai and Vava’u in the case of Tonga). Selection and training of teacher trainers: training the teacher trainers must be done by an expert who has an in-depth knowledge of the content material as well as practical knowledge of how to present this content to adults. In the case of EGR programs, the most suitable teacher trainer has been sufficiently involved in the development of the EGR TG and has previous experience in the training of teachers in the particular context. Under PEARL, the teams of trainers in each country comprised staff from the respective Ministry of Education. In many programs, especially those implemented by specialized research institutions and non-government organizations, the training is facilitated by experts from these organizations. This approach has the benefits of ensuring technical quality in the delivery however, it carries an inherent risk of providing limited capacity building opportunities for staff with mandate over teacher professional development activities be it in government or a teacher education institution. The best approach is heavily reliant on in-country staff with prior experience in the sector who are brought together into the development process of the TG. In this way, the development of the TG serves as an in- depth training on the scope, intensity and interconnections of the TG content with other instructional and assessment elements in the curricula. In-country language and pedagogical experts should also form part of the teacher training team by being a trainer themselves (direct training model) or trainers of officers conducting the teacher training workshops (trainer-of-trainer models). In-country staff can also include a 16 https://bloomlibrary.org 68 specialist from local teacher education institutions who could then be tasked in the future with providing EGR training in pre-service programs. In Tonga, expertise was brought in from Tupou Tertiary Institute, one of the lead providers of tertiary Teaching Studies in the country, also providing Tongan language expertise. It is important that the teacher trainers receive explicit training and preparation for the teacher training workshops. In some cases, there can be an assumption that trainers who have worked together in a task share an equal level of knowledge and understanding about the concepts to be introduced in the teacher training. This cannot be taken for granted. Experience from PEARL showed training activities for teacher trainers through two to three days of workshop-style guided practice helped iron out differences in understanding around key concepts in: the theory of reading; the relative importance of certain skills at particular time of year; when and how formative assessments must be introduced; and the support that can be expected during coaching. In addition, having the expert of teacher trainer training helped the group connect the learning expectations of the content and activities they lead with the overall learning objectives of the training workshop. This proved useful when they could defer a question for another trainer, and in some cases, when they could rely on the examples of analogies presented by co-trainers to make their own delivery clearer. Trainer coordination and teacher-trainer ratios for face-to-face training Trainer coordination is crucial to ensure adequate planning and delivery of teacher training. This is particularly true when multiple training workshops will take place in parallel in different locations. Although teacher training will already be integrated in the EGR implementation plan, it is efficient to have a lead trainer who will be responsible for ensuring trainers’ compliance with the plan, preparation and activities at actual teacher training sessions. Training evaluation and documentation There are different feedback mechanisms that are useful to evaluate the delivery of training. While information about whether teachers are satisfied with the training is useful, training evaluations must assess the workshop’s effectiveness to deliver the content (learning) and the delivery format. On content, training evaluations must identify whether teachers are leaving the intensive training workshops with enough knowledge about how to implement the program in their classrooms. Most informative evaluations use a combination of teacher responses and trainer observations. In Tonga, the teachers filled out a form that provided their own self-assessment on each instructional routine, and their assessment on initial and final levels of knowledge. As part of the last day of the training, trainers assessed teachers in their delivery of a full lesson. All teachers were assessed, and their information compared to their own self-rated assessment of knowledge and competence. Initial discrepancies –between teacher and trainer assessments were identified (eg. a teacher rates themselves positively but the observer rates them as weak), providing useful feedback for discussion during the first coaching session. These measures were collected for all training workshops (one per term, three per year), and helped frame coaching discussions and teacher’s own self-reflections in a journal. Data from the training evaluations in Tonga showed convergence in knowledge and competence in skills already familiar with teachers’ prior experience, e.g. phonics and letter name knowledge, and divergence in activities unfamiliar to them, e.g. oral reading comprehension. The results informed 69 improvements in the delivery of subsequent workshops and identified areas where coaches would have to focus their support in 1-1 visits. This information also informed improvements in the overall logistics of coaching and training planning. Short, peer–level training run by a coach with support of a proficient teacher allowed other teachers in the same school to improve their knowledge of specific instructional routines, allowing a certain degree of standardization to continue in the delivery of intensive training workshops. Classroom observation For teacher professional development, a low-stakes observational assessment of instructional practices allows for the identification of weaknesses in knowledge or pedagogy. It allows for better targeting of teacher training by providing a better understanding of where teachers’ prior knowledge and pedagogical knowledge is strongest and weakest. Observational assessments also help identify the areas of the instructional approach or content of the TG that students interact with most and the least, informing revisions and updates to the instructional approach. A classroom observation is a tool giving educators the opportunity to see teachers in action and to provide objective feedback, beyond opinions, that can help them improve their practice. In EGR programs, classroom observations have been created as a tool to observe teacher’s instructional practices for reading development in early grade classrooms. These observation tools are not intended to be administered as a blank slate to all teachers in the school as instruction across grades varies. In EGR programs, a high-quality classroom observation tool emphasizes the relevant teacher- student interactions and instructional strategies for letter and word-level reading skills, and fluency. It can also provide critical information to assess teacher responsiveness to student queries and interactions and the engagement level by students during the lesson. The experience of various countries developing and using classroom observations in early grade classrooms can be positioned along a continuum ranging from mainly an inventory of resources to quantifiable documentation of instructional routines and skills promoted in a given EGR lesson. Decision on the scope of the classroom observation tool are determined by the context, the objectives of the EGR program, and available capacity to administer and assess the data. In Papua New Guinea (PNG), the Classroom Observation tool used for a reading booster pilot had a strong inventory component and a soft instructional component. The main reason for this was the integration of the EGR pilots within a large-scale program distributing classroom and reading resources nationwide. Moreover, little was known about the actual instructional practices of teachers across provinces. The PNG Classroom Observation aimed at mapping classroom and instructional resources to inform the rollout of project activities, and to explore variation in instructional practices and presence of instructional activities fostering predictors in elementary and lower primary grades. In the case of Tonga, the opposite was true: the Classroom Observation tool was developed to identify the actual delivery of EGR practices as outlined in the Grade 1 and Grade 2 curriculum for Tongan language, and to measure implementation fidelity of the instructional intervention under the “Come, Let’s Read and Write” EGR program (see Annex 7). Staff administering classroom observations are at risk of making subjective inferences around what they observe; therefore, training on the use of the tool is crucial. Training would cover the goal of 70 using the tool, protocol, and how to provide feedback to teachers, in addition to other tasks derived from the use of the tool. Training activities combine a theoretical element (observers practice amongst themselves to master the protocol) and practical element (observers practice in an EGR classroom and then bring their insights for group discussion). Initial training in Tonga was 2.5 days with at least one reinforcement training during the school year. Coaching To compound the effectiveness of teacher trainings, EGR interventions provide ongoing monitoring, feedback, and in-classroom coaching to teachers. Typically, teachers struggle to retain and put into practice new knowledge gained from trainings (Clark-Chiarelli and Louge 2016). It is also common for teachers to only partially follow instructional guides (RTI International 2011). Coaching and continual feedback mollify these problems. Some of the most effective trainings involve modeling from a coach (Abadzi 2006, 127-130). In general, teachers tend to apply more of what they learned when trainings feature in-classroom coaching and feedback (Kim et al. 2016). Coaching can be defined as being onsite, job-embedded, and sustained professional development for teachers. Coaches must have specific expertise to assist individuals and groups of teachers in gaining the knowledge and skills needed to improve instruction (Bean 2014). When teachers are inexperienced or lack knowledge about literacy instruction, a directive model of coaching that promotes fidelity of implementation may be appropriate. Coaching should also provide opportunities for teachers to reflect on their instructional practice and promote teacher learning through modeling and co-teaching activities (Bean 2014). In order to adequately support teachers in their journey to improve their teaching practice, coaches need to spend enough quality time (focused interaction, evidence-based and practical feedback, and positive affirmation) with teachers to make a difference. Group coaching can provide overall information, define a common language for all teachers involved, and establish a positive context for individual coaching. Individual coaching on the other hand, provides the opportunity for a coach to work more intensively with a teacher on program implementation and student learning issues. Just as teachers need support to improve their practice, coaches need adequate support and opportunity to gain the required skills. Coaches must have a deep understanding of the content for which they are responsible. In addition, they must have dispositions that enable them to work effectively with other adults, including well-developed interpersonal, leadership, and communication skills (Bean 2014). Coach training becomes indispensable not only to ensure coaches understand their new roles and responsibilities but to ensure that they do have the depth in knowledge to explain, model, make connections across concepts, and provide practical and constructive feedback to teachers. Additionally, reflection on coaching practice and documentation are important both for understanding overall program implementation, and to inform changes and variations of the coaching component that will be necessary for subsequent stages and/or scaling of the coaching support. Without coaches reflecting on their practice, they may focus too much on how and why teachers are changing without taking the time to reflect on the effectiveness of their own efforts to support the continuous development of their trainees. Human resource constraints in some Pacific countries may require the coaching role to be incorporated into existing school visiting roles, such as supervisors and/or school inspectors. Because effective coaching relies on a positive relationship between the coach and the teacher, coaches who are 71 asked to serve in a supervisory or inspection role may have difficulties developing such relationships. However, by treating teachers with respect and providing coaching activity options, coaches can act in this dual role if necessary. Coaches should be clear with teachers when they are serving in the role of a monitor and when they are providing coaching support. A job description, developed at the initial stage of a coaching program, with provision of training for coaches, is essential for understanding the purposes of the coaching program, the expected roles of the coach, and how coaches are to be evaluated. In Tonga, a team of coaches was initially formed by engaging the reading materials development team, who were also the trainers of teacher training. Each coach visited the teachers they were assigned once every two weeks on the main island, and once a term on the outer islands. As the program matured, two MET district officers were added to the coaching team on the outer islands to provide more regular support to teachers implementing the intervention there. Because of the limitations on human resources, budget and time affecting individual coaching, EGR programs have experimented with various teacher- coach ratios. These vary from as little as 7:1 (Tonga CLRW) to 12:1 (Liberia). Larger ratios are associated with poorer student outcomes. Hybrid models and materials for continuous support Although most coaching is delivered in face-to-face meetings, there have been recent efforts to use technology for communicating with teachers. Combining these two methods into a hybrid model seems to work most effectively, although evidence is still emerging. However, in developing countries, the lack of infrastructure may limit the use of coaching via technology (Bean 2014). In terms of materials, a variety of options may be available to teachers during and at the time of the training workshop to refresh training content. For example, a media compilation of videos that demonstrate the modelling of each of the instructional routines, provide examples and practical demonstration of routines promoting higher- order skills, etc. In PNG, a pilot approach funded by VSO International provided quick tips to teachers via SMS, which appeared to improve overall awareness of sequencing in lessons but no significant effects on student learning. Monitoring and supervision Monitoring In EGR programs, the focus of monitoring activities is to ensure implementation fidelity. Implementation fidelity refers to “the degree to which an intervention or program is delivered as intended” (Snowling, Muter and Carroll 2007). Monitoring implementation fidelity provides crucial information about the treatment itself to support causality claims (Figure 3.4). It also provides valuable information about the implementation challenges that are going to be relevant in decisions to maintain or scale up the program. 72 Figure 3.4: Implementation fidelity as a moderator between project design and outcomes Source: Snowling et al. 2007 Defining the mechanisms and approach to monitor implementation fidelity of EGR programs have important resource implications and are dependent on program objectives and context. To the extent possible, efforts should be made to rely on national systems and processes when determining monitoring activities. Using existing systems benefit from awareness and recognition of the purpose and delivery as people know what they are and how they work. However, monitoring implementation fidelity requires closer follow-up on adherence, i.e. how well the EGR program was implemented as intended in terms of coverage, content, frequency and duration. Moreover, measuring adherence also requires identifying the way in which moderating factors such as the intervention’s complexity, facilitation strategies, quality of delivery and participant responsiveness affect teachers’ ability and disposition to deliver the program. Table 3.1 organizes these terms in relation to EGR programs. Monitoring implementation fidelity is essential to understanding the mechanisms that support the changes in learning outcomes observed at the end of the program. Whether the EGR evaluation renders optimistic, less-than-expected results, or negligible results, without data on implementation fidelity it is important to assess the degree to which results are linked to the EGR program. For example, if there were no effects, was it because the intervention was not effective or because it was not implemented adequately? Were all the aspects of the intervention delivered as intended? Are low levels of implementation related to teacher’s perceiving that the intervention was not useful or “too complex” and irrelevant to them? Without data on implementation fidelity, these questions will remain unresolved and the causality of the EGR program could not be ascertained. In the case of EGR programs, the main factor driving the intervention’s impact comes from improved teacher practice. This is supported by adequate resources, lessons that allow sufficient time- on-task to develop predictor skills, and adequate pedagogical and content support during the implementation. Therefore, the EGR program should account for adequate monitoring protocols that capture changes in teacher’s knowledge, attitudes and behaviors as these become instrumental to improve their practice. A good practice is to ensure that monitoring of implementation fidelity is carried out by a specialist not involved in the EGR program to avoid a potential conflict of interest. In Tonga, external firms were hired to monitor implementation fidelity in years 2 and 3 of the impact evaluation. 73 Table 3.1: Factors determining and moderating teacher adherence in an EGR program EGR resources supporting EGR M&E instruments Element of implementation fidelity EGR Program objective teacher adherence Adherence What to teach? The program includes tested instructional activities that support the EGR Teacher Guide Content development of predictors skills that are relevant to a particular EGR Supplementary Materials EGR Teacher training modules context. EGR Coaching training modules Who to teach? EGR Classroom observation tool Decisions over grades and classrooms to be covered during the EGR Coverage EGR monitoring data systems program and whether all or some schools/classrooms are to be EGR Program Guidelines included. EGR Teacher Guide When and for how long? Frequency & Duration Decisions about when -e.g. time allocated for language and reading class – and how often during the term/school year are included. Moderators Is this intervention too complex to implement? Intervention complexity The more complex or vague an intervention is, the lower its EGR Program Design implementation adherence EGR training evaluation EGR reflection workshop for Are all teachers receiving the same information, training and EGR training guidelines training teacher trainers coaching in essentially the same way? EGR coaching guidelines EGR program implementation Facilitation strategies EGR program administrators must ensure that everyone is receiving EGR guidelines for coaching schedule the same training and support, with the aim that the delivery of the reporting and data collection intervention is as uniform as possible. forms Are teachers implementing all elements of the program according EGR Coach / Monitor Report EGR Teacher Guide to program guidelines? EGR Classroom observation tool Quality delivery EGR Supplementary Materials Poor delivery of the intervention is likely to have an effect on the EGR Teacher Profile [Tonga]* EGR Coaching Guidelines degree to which learning improvements are realised. Do teachers believe and see the benefit and relevance of the EGR program for their practice? EGR Program Guidelines Participant responsiveness The less enthusiastic participants are about an EGR induction information EGR Coach / Monitor Reports intervention, the less likely the intervention is to be implemented EGR Coaching guidelines properly and fully World Bank adaptation, based on Snowling et al. 2007 Supervision Supervision is typically not an area of focus of EGR programs, but it can be key for program effectiveness. Although not an area where most EGR programs implement EGR-oriented activities, supervision can have an impact on the implementation delivery of EGR programs. For example, inspectors and other government officials unclear on the objectives and scope of the program may block EGR activities if they consider they are not aligned with official guidelines. They can also affect participant responsiveness and facilitation strategies if they withhold information about critical resources needed for implementation success. The opposite is also true. When an engaged supervisor (or any other system agent with mandate over supervision activities) becomes an EGR ally, they tend to become a “translator” of information that benefits EGR implementation and awareness efforts, and can play an important role across multiple levels of the system articulating barriers and gaps in the education system that limit the system’s effectiveness to deliver quality education. In Tonga, to cope with limited human resources and substantial travel costs, two district officers in EGR coaching activities in their provinces. As they become better acquainted with their coaching duties, their understanding about the factors limiting teaching and learning of reading in the early grades improve. With this new knowledge, they were able to see the links between program-level requirements and system-level requirements. This recognition allowed them to match barriers to the program with system barriers at the policy level on teacher deployment, promotion, parallel activities that create synergies, and key features of school leadership that could be harnessed to empower school administrator to enhance literacy environments in their schools. Their findings were not only recorded in monitoring documents, but they also got the opportunity to debrief senior management on their findings. Their transformational role can be summarized by a common phrase they used when they brought system-level findings for discussion: “in the end, it’s about helping the schools and the teachers help the children”. Community engagement and awareness (community level) Additional early literacy gains can be made if families and communities can also be enlisted to support children reading. Programs around the world are contributing to improved learning outcomes for students in the early grades of primary school. Attention has been focused on developing materials and training and supporting teachers to improve literacy instruction in the first few years of school. However, a greater understanding is needed of how parents, other adults and children interact during the crucial early years when children are expected to build language and literacy skills (blueTree Group 2014). Initiatives that promote community and parental engagement follow an approach modeled after health communication. In this approach, effective health communication strategies are used to test the impact of behavior change communications on how adults and children interact in relation to language and literacy development, and how those interactions contribute to children learning to read. As explained in Chapter 2, a Social Behavior Change Communication approach (Figure 3.5) was prepared in Tonga to support the “Read With Your Child” public awareness campaign. In this model, key messages were communicated through a systematic communications campaign to identify factors that would promote more reading-related activities (behavioral change) at the individual, community and social levels. Initial impacts (qualitative) show positive responses and initial changes in behavior of adults around reading with young children. The compliment to this was a campaign for beginning readers building on 75 the mechanisms of the first campaign. Figure 3.5: Social Behavior Change Communication Approach Source: FHI 360 2012 C. Monitoring and Measuring EGR programs In monitoring EGR interventions, there are three types of complementary activities that are undertaken regularly for the purpose of informing implementation and tracking progress: (i) monitoring program implementation; (ii) measuring outcomes of the program; and (iii) bringing results to policy. Each has a distinct function and purpose, and neither should be excluded. Monitoring the program implementation Monitoring plays a crucial role in measuring implementation fidelity. Measuring implementation fidelity is crucial to understand when and how the elements of the intervention are driving the changes observed in the classroom and on student learning. In EGR programs, the main agent driving change on student learning outcomes are teachers with their improved instructional practices in the classroom. Interactive, purposeful, intentional and explicit instructional routines, delivered properly, are expected to create the conditions that can engage students in learning to read and write. In this sense, measuring how well the program was implemented and how it motivated these behaviors is essential. The change in behavior is nurtured through teacher training workshops (group training) and coaching visits either in the form of group coaching or individual visits. Monitoring implementation fidelity of training includes measuring that the training was delivered for the intended audience; and that they understand the material of the subject and its pedagogy. Measuring implementation fidelity also requires capturing information on whether the program is being administered as intended in the classroom. For this purpose, EGR programs use classroom observation tools. The observation tools format, dimensions of measure and uses during implementation vary according to context and program objectives. At its basics, the classroom observation should ensure that the behaviors and interactions outlined in the EGR 76 program are being monitored based on the design. In Tonga, the CLRW included a classroom observation tool called Classroom Observation Snapshot (COS) that was used for two complementary purposes. First, as a coaching tool, to ensure coaches focused their feedback on behavior and practices that needed to be present during a CLRW instructional lesson. The COS for Coaching (COS-C) helped coaches provide targeted feedback to their teachers on specific ways to assist the observed teacher improve a routine and the interaction of the routine with others during a lesson. Instructions on how to administer the tool and what the expectations for each routine were directly related to the EGR TG instructions. The COS-C was administered only by coaches for the purpose of measuring implementation fidelity and identifying instructional elements that teachers were struggling with. Features of the program were adequate tool validation, inter-rater reliability across coaches, reflection sessions to understand the goal of using the tool, and use and reporting of COS data. Second, as a monitoring tool, the COS was adapted to explore differences in instructional practices between classrooms in the treatment and control groups. The Classroom Observation Snapshot for Monitoring (COS-M) was similar in structure and instructional expectations to the COS-C. However, where the COS-C had unambiguous expectations on practices and behaviors that needed to happen in the classroom, the COS-M only captured the presence of an instructional activity within the key dimensions of a reading lesson. In the CLRW program, there were explicit instructional practices introduced to achieve official curriculum expectations for Tongan language in Grades 1 and 2. Both treatment and control schools were expected to exhibit instructional practices that helped students achieve the intended learning expectations. In using the COS-M, the presence of the identified instructional practices was “counted” as observed. Thus, a COS-M in a treatment school should have an almost perfect score considering these practices are expected to be observed. In control schools, the presence of any other instructional practice relevant to that instructional dimension was counted as present. Thus, comparisons between treatment and control schools using COS-M data only allow for inferences on the extent to which teachers were using any instructional routine that promotes a particular predictor skill of early grade reading. The COS-M was administered by independent monitors (not associated with the EGR program) trained on the administration of the tool. The adaptation of the tool included an initial exploration workshop to achieve clarity and understanding on what the observable instructional routines and practices should be that promote EGR skills. As part of this workshop, the tool was validated and inter- rater reliability across monitors was completed. Monitors also held reflection before and after the monitoring periods (twice a year) to understand the goal, use and reporting of COS-M data. Overtime, COS-M data helped produce teacher profiles of EGR instruction (Annex 8) that supported in-school debriefs between coaches, teachers and the school principal in treatment schools. Teacher profiles from control schools were used to help curriculum officers and teacher professional development staff in their discussions with teachers and school principals about ways to improve instructional practices. Measuring the program outcomes To improve learning outcomes, the theory of change supporting EGR programs emphasizes the link between effective instruction and better learning. In this sense, EGR programs incorporate summative and formative assessments to measure overall impact (summative) and understand the rate of improvement (formative) during the program. Both assessments fulfill important roles in an EGR program 77 implementation and evaluation. Formative assessments EGR formative assessments are not intended as high-stakes tests, but essentially to provide teachers with information on whether their instruction and the conditions in which it was delivered are affecting student learning as expected. Experience from EGR programs around the world show a variety of formats and times of administration for formative assessments, but they share the characteristics of being introduced at multiple times during the school year, administered by the teacher or with the help of an assistant, and directly linked to instructional practices and skills covered in the EGR TG. In the case of Tonga, the CLRW program created a battery of formative assessments for classroom use, a simplified version of sub-tests of the EGRA tool and modeled after the Annual Status of Education Report reading test used by Pratham. The PRAT was a tool that covered accuracy (not fluency) in four sections: individual sound letters, word reading, reading a sentence and reading a one-paragraph story. Sentence and story reading were followed by two comprehension questions each. The PRAT was administered by coaches in Tonga to reduce the weight of implementing the EGR program on teachers. The coaches kept track of the students tested during the school year and their scores in each of the skills tested, and for the small sample of students that were tested throughout the year, an individual report was produced together with the teacher to be used in reflections with peers in the same school and with teachers. Although many teachers strongly supported the use of PRAT to inform instruction, they recognized the difficulties in administering the tool on their own and for the whole class. To lessen the burden of collecting and documenting results, a digital version of the PRAT (Tangerine-enabled version) was developed that could ease PRAT administration by teachers and produce automatic records for tracking and reporting to peers and parents. Summative assessments Summative assessments are important to determine the improvement on learning outcomes associated with the EGR program. The decision on which tool will be used for summative purposes are determined at the initial stage, when decisions over the evaluation model and rigor required are weighted. EGR programs around the world show a wide range of evaluation modalities from cross- sectional studies to randomized control trials. Each evaluation modality has specific requirements about when outcome measures should be captured. They also have specific requirements (and challenges) regarding group selection, flexibility for maintaining the evaluation design over the program, participant selection, communication, cost, replicability and scalability, among others. In PICs, EGR programs used locally developed, validated EGRA tools to capture improvements in student reading achievements. There were two reasons for this decision. First, the EGRA tool measures all the skills research shows are crucial to help children learn to read and write. Second, in each of the countries supported, there was broad awareness and recognition of the technical merits of the EGRA tool among teachers and government officials and the relevance of the information provided. Teachers and government officials understood what each of the dimensions meant and were able to quickly connect results from sub-tests with curriculum expectations as stated in the official guidelines. The use of EGRA was suitable for use in both an impact evaluation design (Tonga) as well as in cross-sectional designs (Kiribati and Tuvalu). 78 Bringing results to policy Once results of EGR program impacts are available, channels to use them to inform policy can include dissemination events and debriefings to government officials, all stakeholders including service providers and in-country development partners. At this level, the data that’s best suited for reporting is that from summative assessments as it provides the aggregate learning achieved in a given period. The selection of materials for dissemination events is important as well as the selection of format and delivery. In these events, it is important that the key messages around what’s known and not known are clear in order to avoid misattribution, focusing discussion on the improvements and the initial factors that are supporting results and plan actions for next steps. Data from summative and formative assessments have an additional forum: curriculum, pedagogy and assessment divisions. These groups should have the opportunity to reflect on what the data suggest and identify ways in which their own divisions can improve the alignment of EGR education services. In these forums, efforts should be made to avoid the temptation of turning EGR measurement into a high- stakes exercise. EGR measurement is first and foremost a formative activity to help teachers adjust instruction. Decisions on summative policy are to be strategic when it makes more sense in the policy cycle. Finally, the information in government debriefs should also include system-level variables and institutional and/or organizational factors that effectively become barriers to the quality delivery of education outcomes. Costing information is important to understand which factors are driving cost and later on, map cost and outcome data to assess the cost-effectiveness of various approaches. At this level, the “what’s next?” section should include actions that can help government explore ways system -wide barriers could be removed or lessened, and how policies in related areas, e.g. teacher deployment, preservice teacher education, teacher incentive, etc. could be changed, improved to support student learning achievement. D. Conclusions The lessons from implementing these programs are wide and vary depending on context and type of intervention package, duration and commitment to continue or scale. Across all, EGR programs when implemented correctly and completely, the projects can deliver a self-reinforcing package of activities that are coherent for learning (Pritchett 2015). Ensuring that children are exposed to literacy-related activities such as playing with rhymes or looking at books at home will have considerable effects on language development and reading success of children. Evidence shows that despite economic difficulties and other stresses faced by low-income families, many still manage to engage in interactions such as shared picture book reading which are motivated by long-term goals such as ensuring their child’s school readiness. This suggests that a home literacy environment of children from low income families can be enhanced, to good effect. A child’s ability to thrive as a reader depends on language context since this is the vehicle for learning and understanding. The quality of instruction as well as partnerships and collaborations between 79 school leadership and the community at large are other important ingredients for engendering a child’s ability and love for reading. The ability of school actors to organize together to deliver effective literacy opportunities in the classroom and in the school, as well as communities at large, become instrumental in helping children become fluent and confident readers. Creating an instructional process requires technical assistance and knowledge of the language and context. It is crucial that a linguist be included as technical assistance to provide quality assurance to the development of any instructional process, program content, integration teacher guides and other material development. There are benefits to having a single individual to lead these tasks in terms of awareness and in-depth knowledge. However, there are also capacity constraints to take into consideration to avoid the risk of delays in rolling out the program. Effective ERG interventions provide capacity development to teachers. Training must include a series of intensive trainings as well as continuous support primarily in the form of coaching. EGR interventions have used different modalities to train teachers but given the importance of modeled practice and peer-feedback, the face-to-face model is favored over others. To compound the effectiveness of teacher training, EGR interventions provide ongoing monitoring, feedback and in-classroom coaching to teachers. These measures help ensure that teachers correctly apply what they have learned in training. When initiating, developing, implementing, or evaluating a coaching initiative, it is important to consider support available for coaching and how accessible schools and teachers are to coaches. Coaching will not be successful if conducted in isolation or approached as the single solution for changing teacher practices or improving student learning. Well-developed, evidence-based approaches for teaching reading that include coaching are critical elements for success. In addition, specific structural and leadership conditions must be in place or in development. Within government, a review of policy must take place to ensure early grade reading is visible in education policy frameworks – funding, goals, metrics and measure of progress – but also that the system and its structure – leadership—are invested in the effort. The message that “Reading is Important” must be disseminated with a unified voice and be heard loud from the top. EGR programs need to be supervised, monitored and evaluated. Monitoring and evaluation of programs are essential to inform policy and programmatic decisions around EGR curriculum, teacher professional development, coaching and assessment. Supervision has an impact on the implementation delivery of the program. Knowledgeable and engaged supervisors can convey information that benefits EGR implementation and awareness efforts and can play an important role across multiple levels of the system articulating barriers and gaps in the education system that limits the effectiveness to deliver quality education. All in all, improving students’ literacy skills requires systemic efforts, involving stakeholders at multiple levels (see Figure 3.1). As illustrated in this chapter, these involve factors related to student, teachers, family members, those in the community, and in larger contexts (Kim et al 2016). Thus, it is important to incorporate a long-term, cumulative-effects perspective. Changing systems, and habits of learning (students) and teaching (teachers) themselves take time and practice. 80 Chapter 4 – Key considerations for scaling up This chapter presents key considerations for scaling up a school readiness or reading instruction interventions. A. Defining, monitoring and evaluating program objectives Program objectives Defining objectives of a program to operate an intervention at a national level is essential to orientate the efforts and incentives of all relevant actors. That a program objective is needed to orientate a program design is fairly straightforward. The challenge becomes what to specify as the objective. In practice, the term “objective” can vary from “higher order” outcomes reflecting the program’s ultimate social goal, for example improving the reading skills of grade 1 and 2 students, to more immediate results, for example providing grade 1 and 2 teachers with training, materials, and coaching. However, to truly orientate incentives and efforts to achieving the program’s ultimate social goal, defining the higher-order objectives of the program is advantageous (e.g.: reading outcomes). This requires program designers to determine and acknowledge the results-chain and theory of change that link the objectives back to the inputs provided by the program. Higher order objectives require the link between program inputs and outcomes in a program’s design to be evidence-based. Typically, the higher order social objectives of an intervention (e.g.: a child’s reading skills) are not directly malleable by the inputs of a program. For example, government teacher training agencies do not directly control how well children perform on a reading test despite better test scores as being an expected result of better teacher training. For this reason, there is often reluctance on the part of program designers to specify a program’s higher order outcomes as the objective of a program, in part because of the political risk that the higher order outcomes may not materialize. However, this risk of failure is exactly what incentivizes program designers to ensure sufficient evidence exists to link the program’s inputs to higher order outcomes, and it especially requires evidence that the intervention has been adapted correctly to local context. Achieving this may involve a pilot and rigorous evaluation prior to scale-up or integration of an evaluation into the scale-up itself. While this may complicate or delay scale-up, it helps ensure that resources are used effectively and reduces the chance of waste. A classic example is Kenya’s government textbook program: no effect on test scores were found when the program was finally evaluated primarily because the textbooks were far too complicated to be understood by students in many of the country’s schools (Glewwe, Kremer and Moulin 2009). Measuring outcomes Identifying quantifiable higher order outcomes provides an objective metric to evaluate the program. A quantifiable outcome measure is essential politically because (1) the financing required for the program is clearly measurable and (2) the program will compete for public funds; hence, presenting evidence that the program is successful will be crucial in discussions with finance ministry staff. Many aspects of education and child development may not be quantifiable or not quantifiable at a cost that is 81 feasible. However, this does not preclude establishing a quantifiable indicator for the program’s outcome. For example, tests of children’s cognitive function including the Peabody Picture Vocabulary Test (PPVT) would not necessarily be feasible to implement at a nationally representative scale because of the cost— this assessment requires a significant amount of interaction between a highly trained expert and a child. On the other hand, rating-type assessments rely on individuals that have knowledge of a child’s capabilities and can be implemented relatively inexpensively. For example, the eHCI was implemented as a census in several small PICs, sampling all children aged 3 to 5. Rating type assessments of school readiness have been shown to be highly correlated with other, more expensive, measures of child development including the PPVT. As a result, program designers need to be creative in choosing an instrument that is a valid reflection of the objectives of the program but also feasible to implement on a routine-basis. Table 4.1 presents the objectives and outcome measures of the Tongan PEARL interventions. Table 4.1 Outcome measures of the Tonga's CLRW and CPBA interventions 1. School readiness CPBA intervention • School readiness (TeHCI) • Home activities to support school readiness (TeHCI) • Community support for school readiness 2. Reading instruction CLRW intervention • Early grade reading skills (EGRA) • Teachers' implementation of teaching methods Measuring and monitoring implementation The intervention’s results-chain and theory of change provide guidance on what intermediate achievements should be measured and monitored. In Tonga, the reading intervention measured reading skills of children as well as the extent to which teachers were implementing the new teaching methods, number of visits by coaches, training session attendance and how materials were distributed. The school readiness intervention measured child development outcomes but also parenting behaviors, the community’s support for school readiness, parent and child attendance at playgroups, the number of playgroup sessions and the quality of these sessions. Both interventions developed procedures for collecting, storing and reporting implementation data. An important consideration is also how to use the data. For each indicator, the operations manual should specify who the intended recipient of the information is, and what actions he or she is expected to take in response to the information. This specification will also help eliminate unneeded indicators and ensure that the program is able to respond and adapt to implementation challenges that emerge. 82 Evaluation and review strategy In programs that have higher-order and quantifiable outcomes, the crux of program design becomes ensuring that the financing provided to the program ultimately translates into improved outcomes. This observation is not trivial: the “crux” of many national development programs is too often procurement or financial management issues. Programs without higher order objectives are especially prone to this. By contrast, the key to designing a successful program with a higher order, quantifiable outcome measure is ensuring the results chain. This, in turn, requires a suitable evaluation strategy. An intervention that is implemented simultaneously and exhaustively across the entire country or jurisdiction does not provide for a control group with which to evaluate the intervention—a suitable control group is needed in order to provide a credible counterfactual of what would have been the outcomes in absence of the intervention. As a result, program designers would need to have a high degree of confidence that the intervention is effective. Best practice would be to begin the intervention in a randomized-selected, nationally representative sample of schools, if the intervention is new to the country or context. In Tonga, the school readiness intervention and the reading instruction intervention were implemented as pilots and evaluated using a randomized-controlled trial design. The advantage of the randomized design is that there is a fair comparison between school or communities beginning the program and other schools or communities and the outcomes in program schools can be accurately attributed to the program itself. In this way, program designers can be confident that scaling up an intervention will achieve the program’s objectives. Often governments want to implement nation-wide programs immediately or want to select which schools and communities receive the pilot intervention. However, even in pilots that are nationally representative, non-randomized designs run the risk that the measured effects may not represent the true effect at scale-up because the control group is not necessarily a fair comparison group. B. Human resource planning Staff roles and responsibilities Staff roles and responsibilities of a scaled-up intervention would largely reflect those of the pilot and need to be clearly specified. A set of guidelines or operations manual for the program would specify the different staff positions and define clearly their responsibilities and objectives, whom they report to, what information they would report, who would report to them, what information they would receive, what authority they would have to make decisions and how they would be held accountable for their performance. Clearly defining these arrangements are critical for staff members to perform well. Table 4.2 presents the staffing positions used to implement the Tonga PEARL interventions as well as their responsibilities. 83 Table 4.2 Program staff roles and responsibilities 1. School readiness CPBA intervention Volunteer facilitator Lead CPBAs Town officer Provide support to help establish CPBA, organize community committees Community liaison Work with town officer and other community leaders to help establish CPBAs Facilitator trainer Train facilitators Facilitator mentor Mentor facilitators at an on-going basis, collect data about CPBAs Data analyst and report maker Compile CPBA data, produce reports 2. Reading instruction CLRW intervention Teacher trainer Provide training to teachers on the new teaching method Material developer Develop teaching methods suitable for the local language Coach Provide on-going coaching to teachers and classroom observations Assessing and staff capacity Each staff position should also clearly indicate the types of skills an individual would need to succeed in the position. For example, mentors for the volunteer facilitators of the CPBA activities would need to be able to critique facilitators’ work but in a manner that does not alienate or discourage facilitators, especially because they are volunteers. They must have the capability to demonstrate how to provide guidance and support rather than being didactic. Those responsible for training and supervising mentors, and program staff more generally, would need to assess whether staff have the technical but also interpersonal skills needed to achieve the position’s responsibilities. This would also be applied to existing government staff for whom the program implies additional responsibilities for the whole period of program implementation. If existing government staff do not have the skills needed, external staff may be needed. In Kiribati, a proposal to apply the PEARL reading instruction intervention to-scale ensures that senior teachers are paired with coaches for at least one year before becoming coaches themselves. Training The effectiveness of a program relies heavily on how well trained the implementing staff are; however, training and mentoring of staff takes time, especially if a program is being scaled up rapidly. The scale-up plan for the intervention should allow for ample time for training of staff. For example, in an implementation plan to apply a reading intervention nationally in Kiribati, five coaches are paired one-to- one with five teachers for a year, and then these five teachers become coaches themselves. Because of the highly technical nature of coaching and mentoring in both the reading and school readiness interventions implemented by PEARL, sufficient time for training of these staff will be required. Training future teaching forces to master the required skills is essential to avoid interruption between different cohort of teachers and ensure a continuum improvement process throughout system 84 transformation. During the pilot, it’s common that the existing and experienced teachers are involved and be provided in-service professional development to implement the pilot activities. Introducing changes and new initiatives at nationwide scale takes significant time and effort. Therefore, in preparing scaling up plan, it’s crucial for the government to also include strategy for preparation of future teacher generation. The strategy will require changes of both training curriculum and training models to ensure graduates master the required skills well before they enter their profession as teachers. C. Costing Staffing requirements and time To provide an initial estimate for staffing requirements for a scale-up, one approach is to divide staffing needs into variable and fixed components. The variable component depends on the number of beneficiaries, for example, the number of teachers being trained. In the Tongan reading instruction intervention, the number of teachers per coach was seven. An approximate national figure can be attained by multiplying the number of coaches per teacher by the number of teachers. Table 4.3 provides approximate variable staffing needs based on the PEARL interventions in Tonga. To complete a more detailed costing, a plan would have to be developed to including specifying which coaches would visit each school and which mentors would visit each community and based on travel times, how many coaches and mentors would be needed. Other positions would also require similar simulations to arrive at a detailed costing. Table 4.3 Staffing requirements per beneficiary of the Tonga PEARL interventions 1. School readiness CPBA intervention CPBA Coordinator 1 Facilitators per playgroup 2 17 Trainers 7 18 Trainings per year 2-3 Playgroups per mentor 12 Mentor visits per playgroup per year 4 2. Reading instruction CLRW intervention CLRW Coordinator 1 Teachers per trainer 9-10 Teachers per monitor 29 Teachers per coach 7 Transportation costs Both interventions require significant transportation cost due to training for both teachers and 17 Number of trainers was geographically determined. Trainings for facilitators were conducted on three main island groups. There were 3 trainers in the main island of Tongatapu, 2 in Ha’apai and 2 in Vava’u. 18 3 trainings in the first year and 2 after that. 85 CPBA facilitators as well as for teacher coaches and facilitator mentors. Many countries already have in- service teacher trainings nationally so the costs of these activities can be used to estimate the travel costs training with the appropriate adjustments for numbers of teachers and number of facilitators. In addition, most countries conduct periodic school inspections which provide an estimate of cost for the coaching and mentoring components of the program. In Tonga, the initial costing of the school readiness intervention relied on the actual costs of school mentors visiting school under the school-based management program. Material costs The cost of materials, including CPBA starter kits and instructional materials are largely variable costs. An approximation can be obtained by multiplying by the number of classrooms or CPBAs. What would require careful estimation are the expected contributions from communities to support the CPBAs. In the Tongan pilot, communities provided support in various forms to CPBAs including venues, food, play materials among others. Public campaign The cost of a public awareness campaign depends entirely on the available budget. The choice of communications channels – meetings, TV and radio spots, internet etc. – and other variables such as length of campaign, number of products etc. will be determined by the funding. If more funding is available, the campaign can have a broader reach using more communications channels, increase the number of phases of the campaign, and install feedback mechanisms to tailor messaging to later phases of the campaign. These feedback mechanisms are particularly important if there are multiple phases, so should be factored into the budget, but again, the range in cost of obtaining feedback can vary greatly. In Tonga, a local communications firm was hired to design, develop and implement the eight- week campaign. The firm was responsible for identifying a communications strategy for the campaign, including methods of getting the messages across, developing the content for each of the communications channels, making arrangements for ‘media buys’ online, on television, radio, and via mobile phone messaging, and preparing brochures for community gatherings, such as church and community meetings. The firm also arranged a campaign launch event, hosted by a member of the Tongan royal family. Some of the material development required travel to the outer islands to ensure the video content was not urban centric. In addition to the firm costs is the time that needs to be spent initially identifying the issues and ways of addressing them through consultations and liaising with the communications firm on the strategy and content. Financial sustainability After a costing has been completed for the program at national scale, the question becomes whether the government is able to support the program. A typical approach is to calculate what percent of government expenditure from the line ministry that the program represents. Future projections of costs, for example as the number of teachers or students grows, would help assess fiscal sustainability in the longer term. Strengthening learning outcomes can reduce repetition rates and dropout rates which increase the efficiency and reduce cycle costs. However, ultimately the program requires financing by 86 either replacing an alternative activity or from increased revenue or borrowing on the part of government—having a rigorous evaluation method built into the program may be essential for gaining support from the finance ministries. The CPBA intervention relies on revenues from the communities, in this case, in the form of in- kind contributions; how sustainable is this revenue source? What communities are expected to contribute should be carefully estimated and then an assessment should be conducted to understand whether these contributions from the communities are sustainable. This may involve calculating how much volunteer time is required, whether there are seasonal variations in the availability of volunteer time (e.g.: agriculture cycles) and whether poorer communities will be able to support them. D. Legislative, policy and regulatory requirements Public servants Both interventions require either the addition of new public servants or adding new responsibilities to existing public servants which may require issuance of circulars or legislative changes. This includes, for example, asking district education officers to promote the school readiness intervention in communities, or asking town officers to organize CPBAs. This may require amending public service procedures or guidelines, depending on the organization of the human resource policy framework. Government budget Formally entrenching new programs into the government budget (e.g.: as a line item) helps ensure government commitment to the program. A new program would require change to the government budget. It is important that the financing for the program be added formally to the budget, for example by adding a specific line item dedicated to the staffing, materials and travel costs associated with the program. Responsibilities of ministries The programs may lead to changes in the responsibilities of particular line ministries. For example, because the chief point of contact for the CPBAs is communities, it may require that the interior ministry becomes that implementing ministry or that the education ministry receive new authorization to interact with communities. This depends on the organization of a country’s government and may require legislative amendments. Despite the challenges this may pose, it is crucial that the program’s activities be formalized in order for it to be sustainable and to ensure buy-in from the relevant actors. Curriculum Depending on how the curriculum is structure, a reading intervention may require changes. In Tonga as well as PNG, where a similar reading intervention was piloted, the curriculum was broad enough that the interventions’ teaching plans could be implemented while still fulfilling curriculum requirements. This is not always the case, however, as some curricula may specify in significant detail what should be taught when. 87 Regulatory framework Scaling up successful piloted interventions may require introduction of new or updates to existing regulatory frameworks. For example, coaching for teaching reading would require specific tasks, a different set of skills and responsibilities to work with teachers and to use proven coaching and classroom observation instrument. This differs from the traditional inspection roles that involve observing classroom at random. As another example, community-run play activities have never been included in the education sector as a means of improving access to ECD in the pacific island countries; it requires a new policy paper to ensure public funding to sustain this low cost but successful intervention that would benefit many. Flexibility Finally, it is expected that as the program is scaled up or operating for some time, changes will need to be made. For this reason, the enabling policies for the program would be as flexible as possible, for example, avoiding legislation whenever possible. E. Planning and scale-up steps for the PEARL interventions Table 4.4 presents suggested steps for scaling up the PEARL’s both interventions on school readiness (CPBA) and reading instruction (CLRW). The randomized design of both approaches is essential as it creates a fair treatment and control group that provide an unbiased estimate of the effect of the intervention. Non-randomized designs (for example, piloting an intervention in the capital city), may not provide a measure of effect that is representative if extended to the rest of the country, and the program may not achieve the desired objectives as a result. In practice, however, a full randomized scale-up may not be feasible or desired. For example, remoteness, difficult terrain, conflict or natural disasters may prevent certain geographic regions to be excluded from early phases of scale-up; randomizing within regions may mitigate these challenges to some extent, but it may only be possible to evaluate impact in certain areas of a country. Randomized designs also require substantial capacity both to design the evaluation but also to implement and ensure fidelity; in some context, this may not be possible or may require substantial additional cost to increase capacity. Finally, there is often substantial political pressure to roll out an intervention simultaneously to all beneficiaries rather than have any sort of scale-up. This can be mitigated to some extent by (1) setting higher order outcomes as objectives of the program and thus requiring rigorous evaluation, and (2) advocacy and capacity building for government. In many cases, scale-up can be tailored to include a randomized component in a manner that is suitable to the local context. In other cases, alternative evaluation methods may need to be used, including regression discontinuity designs or non-experimental methods; however, these methods all require substantial expertise and even more capacity than randomized design. Ultimately, if a rigorous evaluation is not possible, policy makers must judge whether existing evidence is sufficient to justify public expenditure on scaling up an intervention. 88 Table 4.4a Example of scale-up plan for the reading intervention Phase 1. Feasibility study Cost and fiscal sustainability analysis Phase 2. Intervention design Teacher instruction Lesson plans and teaching materials Teacher training guides and lesson plans Coaching guides Monitoring and evaluation EGRA adaption, instrument development and sample design Sample design Teacher observation instrument development Coaching reporting instruments Other reporting instruments (participation in teacher training, distribution of resources) Data management system and reporting tools Operation manual Guides for program administrators Scale-up plan Legislation / policy / regulation changes Phase 3. Pilot implementation to 20% of schools (e.g.: randomly selected) Preparation of intervention team Training of implementation staff (administrators, etc) Training of coaches Teacher training On-going coaching End of school year EGRA and teacher observations Impact analysis, review of results and revisions to program Phases 4-7 Gradual scale-up to each group of 20% of schools Extension of intervention team Training of additional coaches Teacher training On-going coaching End of school year EGRA and teacher observations Impact analysis, review of results and revisions to program 89 Table 4.4b Example of scale-up plan for the CPBA intervention Phase 1. Feasibility study Cost and fiscal sustainability analysis Phase 2. Intervention design CPBAs Activity guides for facilitators Materials for training facilitators Guides for facilitator trainers Guides for facilitator mentors Guides for community liaisons Start-up kits Monitoring and evaluation School readiness assessment development (EHCI) and workshops CPBA data collection forms Data management system and reporting tools Operation manual Guides for program administrators Scale-up plan Legislation / policy / regulation changes Phase 3. Pilot intervention to 20% of communities (randomly selected) Preparation of intervention team Training of program administrators Training of facilitator trainers Training of facilitator mentors Contact with communities, support to establish CPBAs Training of facilitators On-going mentoring End of year data collection and school readiness assessment Impact analysis, review of results and revisions to program Phases 4-7 Gradual scale-up to each group of 20% of communities Preparation of intervention team Training of facilitator trainers Training of facilitator mentors Contact with communities, support to establish CPBAs Training of facilitators On-going mentoring End of year data collection and school readiness assessment Impact analysis, review of results and revisions to program 90 References Abadzi, Helen. 2006. Efficient learning for the poor: Insights from the frontier of cognitive neuroscience. Washington, D.C.: World Bank Alexander, K.L., Entwisle, D.R., Horsey, C.S., 1997. “From first grade forward: early foundations of high school dropout.” Sociology of Education 70 (2), 87–107 Anderson, Alonzo B. and Stokes, Shelley J. 1984. “Social and Institutional Influences on the Development and Practice of Literacy.” Pp. 24-37 in Awakening to Literacy: The University of Victoria symposium on children’s response to a literate environment: Literacy before schooling, edited by H. Goelman, A. A. Oberg and F. Smith. Victoria, British Columbia: University of Victoria. August, D., and Shanahan, T. 2006. Developing Literacy in Second-Language Learners: A Report of the National Literacy Panel on Language, Minority Children, and Youth. Mahwah NJ USA: Lawrence Erlbaum Associates Azariadis, C. and A. Drazen 1990. “Threshold externalities in economic development.” The Quarterly Journal of Economics. 105 (2): 501-526. Basu, K. and J. Foster 1998. “On measuring literacy.” Policy Research Working Paper Series. No. 1997. Washington, D.C.: The World Bank Bean, Rita M. 2014. The power of coaching: Improving early grade reading instruction in developing countries. Washington, D.C.: USAID Bentaouet Kattan, Raja; Macdonald, Kevin Alan David; Patrinos, Harry Anthony. 2018. Automation and labor market outcomes : the pivotal role of high-quality education (English). Policy Research working paper; no. WPS 8474. Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/356581528983322638/Automation-and-labor- market-outcomes-the-pivotal-role-of-high-quality-education Berlinski, S. and N. Schady (2015). The Early Years: Child Well-Being and the Role of Public Policy. Washington, D.C.: Inter-American Development Bank Black, Maureen; Walker, Susan; Fernald, Lia; Andersen, Christopher; M DiGirolamo, Ann; Lu, Chunling; Mccoy, Dana; Fink, Günther; Shawar, Yusra; Shiffman, Jeremy; E Devercelli, Amanda; T Wodon, Quentin; Vargas-Baron, Emily and Grantham-Mcgregor, Sally. 2016. “Early childhood development coming of age: Science through the life course.” The Lancet. 389. 10.1016/S0140- 6736(16)31389-7. blueTree Group. 2014. Best practices for developing supplementary reading materials. Final report. Washington, DC: United States Agency for International Development (USAID). Retrieved from https://pdf.usaid.gov/pdf_docs/PA00JV69.pdf 91 Brinkman, S., & Stanley, F. 2012. "Child Indicators in Public Health." In A. Ben-Arieh, I. Frones, F. Casas & J. Korbin (Eds.), Handbook of Child Wellbeing: New York: Springer Science + Business Media Dordrecht. Brinkman, S., A. Hasan, H. Jung, A. Kinnell, M. Pradhan. 2015. “The Impact of Expanding Access to Early Childhood Services in Rural Indonesia: Evidence from Two Cohorts of Children.” World Bank Policy Research Working Paper Series. No. 7372. Brinkman, S., & Thanh Vu, B. 2017. Early Child Development in Tonga: World Bank. Washington, DC. Britto, P. R., Lye, S. J., Proulx, K., Yousafzai, A. K., Matthews, S. G., Vaivada, T., . . . Bhutta, Z. A. 2016. "Nurturing care: promoting early childhood development." The Lancet, Published Online October 4. doi: 10.1016/S0140-6736(16)31390-3 Bronfenbrenner, U. 1979. The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press. Bruns, Barbara; Filmer, Deon; Patrinos, Harry Anthony. 2011. Making schools work : new evidence on accountability reforms (English). Human Development ; perspectives. Washington, DC: World Bank. Butler, S.R., H. W. Marsh, M. J. Sheppard, and J. L. Sheppard 1985. “Seven-year longitudinal study of the early prediction of reading achievement.” Journal of Educational Psychology, 77, 349-361 Clark-Chiarelli, N. and N. Louge. 2016. “Teacher quality as a mediator of student achievement.” Pp. 30- 40 in Understanding what works in oral reading assessments. Montreal: UNESCO. Currie, J. and D. Thomas. 1999. “Does Head Start help Hispanic children?” Journal of Public Economics Volume 74, Issue 2: 235-262 Domitrovich, C. E., Morgan, N. R., Moore, J. E., Cooper, B. R., Shah, H. K., Jacobson, L., & Greenberg, M. T. 2013. "One versus two years: Does length of exposure to an enhanced preschool program impact the academic functioning of disadvantaged children in kindergarten?" Early Childhood Research Quarterly, 28(4), 704-713. Duncan, G., & Magnuson, K. 2013. "Investing in Preschool Programs." Journal of Economic Perspectives, 27(2), 109-132. doi: 10.1257/jep.27.2.109 Elango, S., Garcia, J. L., Heckman, J., & Hojman, A. 2015. "Early Childhood Education." IZA Discussion Paper No. 9476: Institute for Study of Labor. Entwisle, D. R., K. L. Alexander, and L. S. Olson. 2005. “First grade and educational attainment by age 22 A New Story.” American Journal of Sociology, 110 1458-1502 EQAP. 2016. “2015 Pacific Islands Literacy and Numeracy Assessment (PILNA).” Suva, Fiji Islands: Educational Quality Assessment Program (EQAP). Retrieved June 20, 2019 (https://www.eqap.org.fj/getattachment/Resources/Final-Regional-Report.pdf.aspx) Family Health International (FHI) 360. 2012. “C-Modules: A learning package for social and behavior change communication.” Durham, NC: Retrieved June 19, 2019 92 (https://www.fhi360.org/resource/c-modules-learning-package-social-and-behavior-change- communication) Farran, S. 2009. Human rights in the South Pacific: Challenges and changes, 181. London: Routledge Cavendish Feinstein, L. 2003. “Inequality in the Early Cognitive Development of British Children in the 1970 Cohort.” Economica, 70 (1): 73-97 Feitelson, D., and Goldstein, Z. 1986. “Patterns of book ownership and reading to young children in Israeli school-oriented and nonschool-oriented families.” Reading Teacher, v39 n9, 924-930. Fernald, L. C. H., Prado, E., Kariger, P., & Raikes, A. 2017. A Toolkit for Measuring Early Childhood Development in Low- and Middle-Income Countries. Washington DC: World Bank Group. Findlay, T. 2011. Gender, Democracy and Multilevel Governance: Early Childhood Development Roundtables in British Columbia. Paper presented at the 2011 Annual Meeting of the Canadian Political Science Association, May. Fischer, K. W. & Silvern, L. 1985. Stages and individual differences in cognitive development. Annual Review of Psychology 36, 613-648. Fisher, D. and Frey, N. 2008. Better Learning Through Structured Teaching: A Framework for the Gradual Release of Responsibility. Alexandria, Virginia: Association for Supervision and Curriculum Development. Gertler, P., Heckman, J., Pinto, R., Zanolini, A., Vermeersch, C., Walker, S., Chang, S. M., … Grantham- McGregor, S. 2014. “Labor market returns to an early childhood stimulation intervention in Jamaica.” Science (New York, N.Y.), 344(6187), 998-1001. Glewwe, Paul; Kremer, Michael; Moulin, Sylvie. 2009. “Many Children Left Behind? Textbooks and Test Scores in Kenya.” American Economic Journal: Applied Economics. 1. 112-35. 10.1257/app.1.1.112. Gregory, T., Harman-Smith, Y., Sincovich, A., Wilson, A., & Brinkman, S. 2016. It takes a village to raise a child: The influence and impact of playgroups across Australia. Telethon Kids Institute, South Australia. Griffen, V. 2006. Gender Relations in Pacific cultures and their impact on the growth and development of children, ‘Children’s Rights and Culture in the Pacific’ Seminar, 30th October 2006 Hambleton, R. K., Merenda, P. F., & Speilberger, C. D. 2005. Adapting educational and psychological tests for cross cultural assessment. Mahwah, NJ: Lawrence Erlbaum Associates Inc. Heckman, J. J., and D. V. Masterov. 2007. “The Productivity Argument for Investing in Young Children.” Applied Economic Perspectives and Policy. 29(3): 446–493. Huffer, E. 2006. Cultural Rights in the Pacific – What they mean for Children, ‘Children’s Rights and Culture in the Pacific’ Seminar, 30th October 2006 93 Janus, M., Brinkman, S., & Duku, E. 2011. "Validity and psychometric properties of the Early Development Instrument in Canada, Australia, United States and Jamaica." Social Indicators Research, 103(2), 283-297. doi: 10.1007/s11205-011-9846-1 Jimerson, S., Egeland, B., Sroufe, L.A. and Carlson, B. 2000. “A prospective longitudinal study of high school dropouts: examining multiple predictors across development.” Journal of School Psychology 38 (1), 525–549 Kerwin, J. T and R. Thornton. 2015. “Making the Grade. Understanding what works for teaching literacy in rural Uganda.” PSC Research Reports No. 15-842, University of Michigan: Population Studies Center Lucas, A. M., P. J. McEwan, M. Ngware, M. Oketch. 2014.” Improving Early-Grade Literacy in East Africa: Experimental Evidence from Kenya and Uganda.” Journal of Policy Analysis and Management. 33 (4) Macdonald, K. A. D. and Vu, B. T. 2018. “A randomized evaluation of a low-cost and highly scripted teaching method to improve basic early grade reading skills in Papua New Guinea (English).” World Bank Policy Research Working Paper Series No. 8427. Washington, D.C. : World Bank Group Macdonald, Kevin Alan David; Brinkman, Sally Anne; Jarvie, Wendy; Machuca-Sierra, Myrna; Mcdonall, Kristen Andrew; Messaoud-Galusi, Souhila; Tapueluelu, Siosiana; Vu, Binh Thanh. 2018. Intervening at Home and Then at School : A Randomized Evaluation of Two Approaches to Improve Early Educational Outcomes in Tonga (English). Washington, D.C. : World Bank Group Marteleto, L., D. Lam, V. Ranchhod, 2008. “Sexual behavior, pregnancy, and schooling among young people in urban South Africa.” Studies in Family Planning 39 (4), 351–368. Mason, J.M., and McCormick, C.E. 1981. “An investigation of prereading instruction from a developmental perspective: Foundations for literacy.” (Tech. Rep. No. 224). Urbana-Champaign, IL: University of Illinois Center for the Study of Reading. Melhuish, E. C., Phan, M. B., Sylva, K., Sammons, P., Siraj-Blatchford, I., & Taggart, B. 2008. "Effects of the Home Learning Environment and Preschool Center Experience upon Literacy and Numeracy Development in Early Primary School." Journal of Social Issues, 64(1), 95-114. Miller, W.H. (1969). “Home prereading experiences and first-grade reading achievement.” The Reading Teacher, 22, 641-645. Montenegro, Claudio E. and Patrinos, Harry Anthony. 2014. “Comparable estimates of returns to schooling around the world (English).” Policy Research working paper; no. WPS 7020. Washington, DC: World Bank Group Nakajima, N., Hasan, A., Jung H., Brinkman S., Pradhan M., Kinnell A. 2016. “Investing in School Readiness: An analysis of the Cost Effectiveness of Early Childhood Education Pathways in Rural Indonesia.” World Bank Policy Research Working Paper Series No. 7832. National Institute for Child Health and Human Development. 2000. Report of the National Reading 94 Panel. Teaching Children to Read: An Evidence-based Assessment of the Scientific Research Literature on Reading and its Implications for Reading Instruction. (NIH Publication No. 00-4754). Washington, DC: National Institutes of Health Naudeau, S., N. Kataoka, A. Valerio, M. J. Neuman, L. Kennedy Elder (2010). Investing in Young People. An Early Childhood Development Guide for Policy Dialogue and Project Preparation. Conference Edition. Washington, D.C.: The World Bank. Naudeau, Sophie; Kataoka, Naoko; Valerio, Alexandria; Neuman, Michelle J. ; Elder, Leslie Kennedy. 2011. Investing in young children: an early childhood development guide for policy dialogue and project preparation. Washington, DC: World Bank Neuman, Susan B. and Dickinson, David K. 2011. Handbook of Early Literacy Research. Volume 3. New York, NY: The Guildford Press. Ninio, A. (1980). Picture-Book Reading in Mother-Infant Dyads Belonging to Two Subgroups in Israel. Child Development, 51, 587-590. http://dx.doi.org/10.2307/1129299 Nores, M. and W. S. Barnett. 2010. “Benefits of early childhood interventions across the world: (Under) Investing in the very young.” Economics of Education Review. 29:271-282. O’Rourke, K., L. Howard-Grabman, and G. Seoane. 1998. “Impact of Community Organization of women in Perinatal Outcomes in Rural Bolivia.” American Journal of Public Health. 3 (1): 9-14 OECD. 2001. Starting Strong. Early childhood Education and Care. Paris: OECD https://doi.org/10.1787/25216031 OECD. 2006. Starting Strong II, Early Childhood Education and Care. Paris: OECD https://doi.org/10.1787/9789264035461-en OECD 2016. PISA 2015 Results. Excellence and equity in education. Volume 1, p.162. Paris: OECD. Parackal, M., Parackal, S., Eusebius, S., Mather, D. 2017. "The Use of Facebook Advertising for Communicating Public Health Messages: A Campaign Against Drinking During Pregnancy in New Zealand." JMIR Public Health Surveill 2017;3(3):e49. DOI: 10.2196/publichealth.7032) PASEC 2015. PASEC 2014. Education System Performance in Francophone Sub-Saharan Africa. Competencies and learning factors in primary education, pp. 36,50. Dakar: CONFEMEN Patrinos H A and Psacharopoulos G. 2010. “Returns to Education in Developing Countries.” pp. 305-312 in: International Encyclopedia of Education volume 2, edited by Penelope Peterson, Eva Baker, Barry McGaw. Oxford: Elsevier Payne, Adam C., Whitehurst, Grover J., and Angell, Andrea L. 1994. “The Role of Home Literacy Environment in the Development of Language Ability in Preschool Children from Low-income Families.” Early Childhood Research Quarterly 9, 427-440. Pianta, R.C. and S.J. Mcoy. 1997. “The First Day of School: The Predictive Validity of Early School Screening.” Journal of Applied Developmental Psychology 18 (1): 1-22. 95 Piper, B. & Korda, M. 2011. Egra plus: Liberia. program evaluation report. RTI International. Piper, B., Zuilkowski, S. S. & Mugenda, A. 2014. “Improving reading outcomes in kenya: First-year effects of the primary initiative.” International Journal of Educational Development 37, 11–21. Piper, B., Sitabkhan, Y., Mejía, J., and Betts, K. 2018. “Effectiveness of Teachers’ Guides in the Global South: Scripting, Learning Outcomes, and Classroom Utilization.” RTI Press Publication No. OP- 0053-1805. Research Triangle Park, NC: RTI Press. https://doi.org/10.3768/rtipress.2018.op.0053.1805 Pressley, M. 1998. Reading Instruction That Works: The Case for Balanced Teaching. New York: The Guilford Press Pritchett, Lant. 2015. “Creating education systems coherent for learning outcomes: Making the transition from schooling to learning.” Research On Improving Systems of Education. RISE-WP- 15/005. Prost, A., T. Colbourn, N. Seward, K. Azad, A. Coomarasamy, A. Copas, T. A. J. Houweling, E. Fottrell, A. Kuddus, S. Lewycka, C. MacArthur, D. Manandhar, J. Morrison, C. Mwansambo, N. Nair, B. Nambiar, D. Osrin, C. Pagel, T. Phiri, A. Pulkki-BrännstrÖm, M. Rosato, J. Skordis-Worrall, N. Saville, N. S. More, B. Shrestha, P. Tripathy, A. Wilson, A. Costello. 2013. “Women’s groups practising participatory learning and action to improve maternal and newborn health in low- resource settings: a systematic review and meta-analysis.” Lancet, 381: 1736-46 Pryor, John, Akyeampong, Kwame, Westbrook, Jo and Lussier, Kattie. 2012. “Rethinking teacher preparation and professional development in Africa: an analysis of the curriculum of teacher education in the teaching of early reading and mathematics.” Curriculum Journal, 23:4, 409-502, DOI: 10.1080/09585176.2012.747725 Psacharopoulos, George; Patrinos, Harry Anthony. 2018. “Returns to investment in education : a decennial review of the global literature (English).” Policy Research working paper; no. WPS 8402. Washington, D.C. : World Bank Group Raikes, A., Yoshikawa, H., Britto, P. R., & Iruka, I. 2017. "Children, Youth and Developmental Science in the 2015–2030 Global Sustainable Development Goals." Society for Research in Child Development Social Policy Report, 30(3). Rao, N., Sun, J., Chen, E., & Ip, P. 2017. "Effectiveness of early childhood interventions in promoting cognitive development in developing countries: A systematic review and meta-analysis." Hong Kong Journal of Paediatrics, 22, 14-25. Raz, I. T., and Bryant, P. 1990. “Social background, phonological awareness and children’s reading.” British Journal of Developmental Psychology, 8, 209-225. doi:10.1111/j.2044-835X.1990.tb00837 Reynolds, A.J., J. A. Temple, D. L. Robertson, and E. A. Mann. 2001. “Long Term Effects of an Early Childhood Intervention on Educational Achievement and Juvenile Arrest: A 15-Year Follow-Up of Low-Income Children in Public Schools.” JAMA. 2001;285(18):2339–2346. doi:10.1001/jama.285.18.2339 96 Rosenzweig, M. R. 1995. “Why are there returns to schooling?” The American Economic Review. Vol. 85, No. 2: 153-158 RTI International (2011). Transforming reading instruction [handout]. Research Triangle Park, NC: RTI International. Schweinhart, L. J., J. Montie, Z. Xiang, W. S. Barnett, C. R. Belfield, M. Nores 2005a. The High/Scope Perry Preschool Study Through Age 40. Summary, Conclusions, and Frequently Asked Questions. Ypsilanti, MI: High/Scope Press. Schweinhart, Lawrence J., Jeanne Montie, Zongping Xiang, W. Steven Barnett, Clive R. Belfield, & Milagros Nores, 2005b. Lifetime Effects: The High/Scope Perry Preschool Study Through Age 40. Ypsilanti, MI: High/Scope Press. Snow, C.E., Burns, M.S., and Griffin, P. 1998. Preventing Reading Difficulties in Young Children. Washington, D.C.: National Academy Press Snowling, MJ., Muter, V., Carroll, J. 2007. “Children at family risk of dyslexia: a follow-up in early adolescence.” J Child Psychol Psychiatry. 2007 Jun;48(6):609-18. Stanovich, K. E. 1986. Matthew Effect in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly Fall. Toganivalu, D. 2008. Early Childhood Care and Education in the Pacific: Reflections of our past, our present and our future. In Puamau, P and Pene F. (Eds.) Early Childhood Care and Education in the Pacific: The PRIDE Project. Suva, Fiji: Institute of Education, University of the South Pacific. Valerio, Alexandria; Sanchez Puerta, Maria Laura; Tognatta, Namrata Raman; Monroy Taborda, Sebastian. 2016. Are there skills payoffs in low- and middle-income countries?: Empirical evidence using STEP data (English). Policy Research working paper; no. WPS 7879; WDR 2018 background paper. Washington, D.C.: World Bank Group. Wells, G., Barnes, S., & Wells, J. 1984. Linguistic influences on educational attainment. Final report to the Social Science Research Council, University of Bristol, Department of Education and Science. Wells, G. 1985. Language development in the preschool years. (pp. 281-320). New York: Cambridge University Press. World Bank. 2012a. How well are Tongan children learning to read? Washington, D.C.: The World Bank World Bank. 2012b. How well are Ni-Vanuatu children learning to read in English? Washington, D.C.: The World Ban World Bank. 2012c. How well are Ni-Vanuatu children learning to read in French? Washington, D.C.: The World Bank World Bank. 2013. New Perspectives on Strengthening Government Capacity to Intervene for School Readiness in Samoa, Tonga and Vanuatu. Washington, D.C.: The World Ban World Bank. 2014a. East New Britain (ENB) Early Grade Reading Assessment (EGRA) Survey. 2012 Diagnostic Results Report. Washington, D.C.: The World Bank 97 World Bank. 2014b. Madang Early Grade Reading Assessment (EGRA) Survey. 2011 Diagnostic Results Report. Washington, D.C.: The World Bank World Bank. 2014c. National Capital District (NCD) Early Grade Reading Assessment (EGRA) Survey. 2012 Diagnostic Results Report. Washington, D.C.: The World Bank World Bank. 2014d. Western Highlands Province Early Grade Reading Assessment (EGRA) Survey. 2013 Diagnostic Results Report. Washington, D.C.: The World Bank World Bank. 2018. World Development Report 2018 : Learning to Realize Education's Promise. Washington, DC: World Bank World Bank. 2019. Tonga’s school readiness and early grade reading pilots: impacts and the case for scale-up. Policy Brief. Manuscript submitted for publication. Washington, DC: World Bank. Wren, Sebastian. 2009. The cognitive foundations of learning to read: A framework. Austin, TX: Southwest Educational Development Laboratory. Young, Mary Eming. 2007. Early Child Development, From Measurement to Action : A priority for Growth and Equity. Washington, D.C.: World Bank. Young, M.E. and F. Mustard 2007 Brain development and ECD: A case for investment. In Africa’s Future, Africa’s Challenge: Early Childhood Care and Development in Sub-Saharan Africa, ed. M. Garcia, A. Pence and J. L. Evans. Washington, D.C.: The World Bank 98 Annexes Annexes can be found in Supplementary Annexes Document Annex 1 – Tonga CPBA Fact sheets and Recipe cards Annex 2 – Tonga CPBA Monitoring Forms Annex 3 - Key elements of the Tonga “Read with your Child” public awareness campaign Annex 4 – Tonga ECDE Advisory Council Guidelines Annex 5 – TeHCI Instrument Annex 6 – TeHCI Scoring Manual Annex 7 – Classroom Observation Tool for CLRW Program Annex 8 – Teacher Profiles 99