Policy Research Working Paper 9061 Built to Last Sustainability of Early Childhood Education Services in Rural Indonesia Amer Hasan Haeil Jung Angela Kinnell Amelia Maika Nozomi Nakajima Menno Pradhan Education Global Practice November 2019 Policy Research Working Paper 9061 Abstract This paper studies the sustainability of early childhood provided. Centers that provided higher quality care, had education centers established under a large-scale, donor- more complementary services in the area, and had more funded project in rural Indonesia. Analysis of quantitative parental involvement were significantly more likely to be and qualitative data shows that 86 percent of the centers sustained after donor funding ended. In contrast, centers continued to provide preschool services three years after with more substitute services in the area were less likely to project funding ended. Centers balanced the reduction in be sustained. There is no evidence to suggest that distance to funding by introducing student fees. The paper estimates a the village center or nearest neighboring center was a major series of logistic regression models to predict center sustain- factor for sustainability. There is also no evidence to suggest ability. Centers that increased their share of expenditures that, while they were operating, closed centers catered to on teacher salaries during the project were significantly children from different wealth backgrounds than those that more likely to remain open. Often this was made possi- remained open. These results point to actionable lessons for ble by centers altering their mix of supplementary services the design and sustainability of future development projects. This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at ahasan1@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Built to Last: Sustainability of Early Childhood Education Services in Rural Indonesia Authors* Amer Hasan (World Bank) Haeil Jung (Korea University) Angela Kinnell (University of Adelaide) Amelia Maika (Gadjah Mada University) Nozomi Nakajima (Harvard University) Menno Pradhan (University of Amsterdam and VU University) JEL Codes D02, D23, F35, I25 Keywords Sustainability, Early Childhood Education, Development Aid, Quality of Service Delivery *Authors are listed alphabetically.   Acknowledgments We would like to thank Dedy Junaedi, Upik Sabainingrum, Anas Sutisna, Lulus Kusbudiharjo and Mulyana for managing the fieldwork and Atsuko Muroga for research assistance. We are grateful to Sophie Naudeau, Alaka Holla and Halsey Rogers for comments on an earlier version of this paper. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Funding This work was supported by the Strategic Impact Evaluation Fund (TF0A0234) at the World Bank. Haeil Jung acknowledges that this work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2016S1A3A2924956). Declarations of interest None.   I. Introduction As the international community adopts and implements the Sustainable Development Goals (SDGs), a growing concern is whether interventions established by donor funding are sustainable. As an illustration, between 2001 and 2013, the World Bank invested USD 3.3 billion in early childhood development through 273 investment projects (Sayre et al. 2015). This figure jumped to roughly USD 1.1 billion per year between 2014 and 2017.1 Although investments in early childhood education have increased, we know very little about whether and how these investments are sustained. A review of the literature suggests that as few as 36% of projects include goals related to sustainability (Chapman and Moore 2010). In this paper, we document the sustainability of a large-scale development project in the world’s fourth largest country – Indonesia. Covering 50 districts across the country, the project ran from 2009 to 2013. In this time it increased access to early childhood education and development (ECED) services in 3,000 villages for over 634,000 children (World Bank 2014) by providing block grants to villages to establish up to two playgroups, providing teacher training and raising community awareness about the importance of ECED. Villages included in the project were selected on the basis of their poverty rates, having large populations – particularly of children in the target age-range for the project playgroups (age 3-4) – and community interest in supporting these services. 2 On average, the sample of households targeted by the project was broadly representative of the rural population in Indonesia at the time.3 We revisited villages in 2016 – 3 years after the project had closed and project funding had ended – to assess whether project services had been sustained. We define sustainability as the continued use of project components beyond their initial funding period (Scheirer and Dearing 2011; Shediac-Rizkallah and Bone 1998; Bamberger and Cheema 1990). Specifically, we measure sustainability of the early childhood education project in two ways: (i) whether there is continued delivery of early childhood education after external funding ended (i.e. whether project playgroups were still in operation) and (ii) whether there is continued employment of teachers trained under the project after external funding ended. This paper uses a unique panel data set on playgroups established under the project. Data were collected at the beginning of the project in 2010, at the end of the project in 2013, and three years after project funding ended in 2016. We focus on the sustainability of this early childhood education project given the growing importance of increasing investments around the globe in early human capital development and the importance of early childhood education opportunities in promoting female labor force participation. Under the SDGs, the global community has committed that by 2030 “all girls and boys [will] have access to quality early childhood development, care and pre-primary education so that they are ready for primary education” (United Nations 2015). However, the global gross enrollment ratio in pre-primary education is still low at 44% (UNESCO 2015), which suggests the need for greater investments in early childhood education over the next decade. As pre-primary education is generally not included in compulsory education, much of this investment will likely come from project-type interventions given that the majority (over 53%) of global disbursements                                                              1 Investing in Every Child’s Early Years: World Bank Contributions. Accessed on August 12, 2019. Available at: https://results.org/blog/investing_in_every_childs_early_years_world_bank_contributions/. 2 For details refer to Appendix 4 in Hasan, Hyson and Chang, 2013. 3 A comparison of assets ownership by households in the evaluation sample with that of the rural sub-sample of the SUSENAS (a nationally representative household survey) suggests average rates of asset ownership and education levels are by and large similar between the two. 1 of Official Development Assistance to developing countries in the education sector has historically been, and continues to be, delivered in the form of project-type interventions (OECD 2017).4 As such, researchers and practitioners alike are seeking to understand whether and how donor funding in early childhood education can be sustained. We examine a range of factors that are often considered important during the project design stage. Do complementary services exist? Are substitute services available? Are the services being offered by the project located conveniently for the beneficiary community? Are they of sufficiently high quality to attract users? Often these considerations are at odds when project designers are seeking to allocate a fixed project budget to various aspects of its design. Understanding which factors matter for continuity of services beyond the period of donor funding can be helpful at the design stage for those considering such choices. The literature on sustainability of social services is spread out across a variety of disciplines. In order to assess what factors have previously been identified as particularly important for sustainability of services, we reviewed articles in the areas of health, education, child care and international economic development. What is clear when reviewing the literature on interventions that are sustained beyond their funding period is that none examines multiple potential predictors of sustainability at the same time. In fact most papers examine the impact on sustainability of just one potential predictor. We used insights from each of the papers across the disciplines listed above to sharpen the focus of this paper and provide a basis for exploring factors that may predict sustainability in our early childhood setting. Specifically, we focus on key variables across domains that include finance, quality of services, the presence of substitute and complementary services, and community support as potential predictors of the sustainability of project playgroups. In terms of finance, previous research shows that a key challenge for sustainability is securing post-project funding. For example, in a literacy project in Zambia, the withdrawal of donor support left primary schools with erratic and unstable funding with which to continue the program (Meki Kombe and Herman 2017). In addition to the need to secure post-project funding, the source of that funding as well as how it is spent can be important predictors of sustainability. For example, childcare centers that received financial support from the government and paid higher wages to staff were less likely to close in a large-scale study of child care centers in Canada (Kershaw et al. 2005). Likewise competition in rural markets has been found to be an important predictor of the survival of rural hospitals in the United States (Succi et al. 1997). In our study we extend this to look not only at competition from substitute services (other playgroups) but also the availability of complementary services in villages (kindergartens). Especially in competitive markets, quality is likely to be a strong predictor of sustainability because, in this setting, parents will not keep their children in poor quality care. Prior research has shown that quality is a key predictor of sustainability. For example, child care centers were significantly more likely to remain open when teachers had higher qualifications as measured by education and teaching experience (Kershaw et al. 2005, Lam et al. 2013). We also examine the link between sustainability and the provision of supplementary services based on evidence from the health sector. Studies have shown that rural hospitals offering a range of medical services relative to other neighboring hospitals have significantly lower risk of closure because they can differentiate themselves from other hospitals (Succi et al. 1997).                                                              4 Amount refers to gross disbursements categorized as official development assistance received by developing countries in constant 2015 prices. 2 Lastly, the education policy literature reports the importance of community support in sustaining reform efforts. In particular, support from local stakeholders is a strong positive predictor of successful institutionalization of school reforms (Bryk et al. 2010). This paper considers each of these factors together rather than individually. As such, it contributes not only to the literature on sustainability of development projects but also offers operational lessons for project designers. We leverage rich panel data on playgroups in rural Indonesia to demonstrate how characteristics of the playgroups during the funded project period predict their sustainability several years after donor funding ends. This is an improvement from existing research on sustainability in health and education, which typically uses cross-sectional data (Scheirer and Dearing 2011). While the focus of this paper is on the sustainability of services established under an early childhood education project, the findings also provide lessons and insights for designing other types of community-based development projects. We present six key findings in this paper. First, we show that the vast majority (86%) of playgroups established under the project remained open three years after project funding ended. In particular, those that remained open balanced the reduction in donor funding by introducing student fees. Both our survey data and qualitative interview data show that the playgroups that closed were those that struggled to find the financial and human resources needed to continue operating. Second, we show that the playgroups that allocated a larger share of expenditure to teacher salaries during the project period were significantly more likely to remain in operation after the project ended. We find evidence that playgroups reallocated their resources towards teacher salaries by cutting down on supplementary services such as weekly food programs. Third, we find that the quality of the project playgroups – as measured by direct observation of the playgroup – is a key predictor of sustainability. We find that such observations capture elements of the quality of these playgroups that are predictive of sustainability, over and above what can be predicted from teachers’ education and teaching experiences alone. Fourth, we argue that the presence of substitutes and complements in the market matters. We find that the number of kindergartens per 100 children (a complementary service, for an older age group) supports playgroup sustainability. In contrast, while the number of playgroups per 100 children (a substitute service) is negatively correlated with sustainability, the point estimates are not statistically significant. We do not find that distance to the village center or to the nearest neighboring playgroup were predictive of sustainability in our setting. Fifth, the data suggest that support for early childhood education in the local community is important for sustainability. We find that the degree of parental involvement in the playgroups is a predictor of sustainability. Finally, when we jointly examine the categories of predictors (finance, quality, market conditions, supplementary services, and community support), finance, local market conditions and supplementary services come out as significant predictors. Local market condition is the strongest predictor category of a center remaining open three years after the project ended. For higher degrees of sustainability, the provision of supplementary services is the strongest predictor category. The paper proceeds as follows. Section II describes the Indonesia Early Childhood Education and Development Project. Section III introduces the data and presents descriptive statistics. The empirical strategy is described in section IV. Section V presents the regression 3 results and section VI concludes with a discussion of the implications of our findings for future work. II. Indonesia Early Childhood Education Project Between 2009 and 2013, the Government of Indonesia invested USD 67.5 million in the Indonesia Early Childhood Education and Development (ECED) Project in partnership with the World Bank. The project aimed to improve “poor children’s overall development and readiness for further education, within a sustainable quality early childhood education system.”5 In 2007, when the project was launched, the investment decision was a significant one. As in most other countries at the time preschool services were not part of the compulsory education system.6 Indonesia had very low preschool enrollment rates, the gross enrollment rate in early childhood education for children ages 4-6 was only 23%, with significant gaps in enrollment across wealth, geographic location, and gender (Jung and Hasan 2016). Government expenditure in early childhood education accounted for only 0.45% of the public education budget whereas 80% was spent on primary and secondary education (World Bank 2014). The project was seen as a way to increase government investment in ECED, to develop a quality assurance system for ECED services and to build on the limited capacity at the district level to implement ECED programs. Under the project, communities received the following package of interventions. First, a community facilitator helped raise community awareness on the importance of early childhood education. These facilitators also provided communities with training on how to prepare proposals for the block grants available through the project. Second, block grants for three years, totaling USD 18,000 per village were spent on establishing up to two ECED centers. Thus, villages received USD 3,000 per center per year over three years. Villages used the block grants to establish playgroups, which are early childhood education programs intended to cater to children ages 3-4. The communities generally provided the site for the playgroups while the grants were used to buy materials, pay teachers, and provide supplementary services. These centers emphasize learning through play before children are old enough to enroll in kindergarten – typically at ages 5 and 6.7 We refer to the playgroups established under the ECED Project as project playgroups. Third, 200 hours of teacher training were provided for up to two teachers per project playgroup. This training was delivered in two blocks of 100 hours each. As noted in Brinkman et al. (2017), the project playgroups operated three times a week for 3 hours a day and allowed children between the ages of 3 and 6 to enroll. Project documents suggest that the cost per child was about US$27 per year.8 This estimate excludes any voluntary                                                              5 The project development objective in the World Bank project appraisal document: http://projects.worldbank.org/ P089479/early-childhood-education-development-project?lang=en. 6 This is still true today. 7 While several types of early childhood services exist in Indonesia, the two most common types of pre-primary education are playgroups (Kelompok Bermain, KB) under the Ministry of Education and Culture, and kindergartens, which refer to both kindergarten (Taman Kanak-kanak, TK) under the Ministry of Education and Culture and Islamic kindergarten (Radhatul Athfal, RA) under the Ministry of Religious Affairs. In contrast to playgroups, kindergartens cater to children ages 5-6 and focus on preparing children for primary school. 8 This is calculated by dividing total project costs for implementation of the community-based component (US$54 million over 3 years) by the actual number of children (673,162 children) reported to have enrolled in the 3,000 villages where the program operated. This information is drawn from the Implementation Status and Results (ISR) Report no. 11 of the project. This is available online at http://documents.worldbank.org/curated/en/684441468267567691/pdf/ISR-Disclosable- P089479-12-29-2013-1388324682405.pdf. All costs are per child per year and in 2014US$. 4 contributions from the villages to the project. Villages often made available the land on which project playgroups were housed. In contrast, other early childhood programs range in cost from US$37 per child in India to US$52 per child in Mexico to US$66 per child in Brazil—suggesting that this package was slightly less costly (Barnett 1997; Evans, Myers and Ilfeld, 2000). Teachers who worked in the project playgroups were predominantly women from the village who often had children of their own. Some had prior work experience in health and education. Others had no such prior experience (Hasan, Hyson and Chang, 2013). Taken together, the package of interventions under the Indonesia ECED Project helped finance the demand for and supply of playgroups in a sample of villages that are broadly representative of rural Indonesia (Hasan, Hasan and Chang, 2013, Appendix 9). III. Data This paper uses data on project playgroups located in villages that received the block grant and were part of an impact evaluation of the Indonesia ECED Project (N=432). 9 All of these playgroups were surveyed at baseline in 2010. In 2013, one project playgroup per village was sampled for follow up (N=245). The rationale for the selecting these 245 for follow up was that several of the children we had been following for the impact evaluation were enrolled in them.10 This was the last survey conducted while the project was still active. In 2016, three years after the project had closed, we conducted phone interviews with administrators of all 432 project playgroups to confirm whether they were still open or had closed. If the playgroup was closed, we visited the village and interviewed former center administrators and teachers to collect data on the date of closure, key reasons for closure, as well as what happened to the building, toys, and teachers after the project playgroup closed. If the project playgroups were still open we conducted interviews to collect information on current operations, teachers, and funding modalities.  Main analytic sample However, the main analytic sample in this paper is the 245 project playgroups that were interviewed in 2013 and followed up in 2016. We restrict our analysis to these panel project playgroups primarily because we have detailed data on the characteristics of these project playgroups at two time points - in 2010 and 2013 as well as the knowledge that they were still operating in 2016 - whereas detailed follow up was not completed for the remaining playgroups in 2013. As a result, we have granular data on sustainability for these 245 playgroups - for each, we not only know whether the it was open or closed in 2016 but we also know the number of teachers who received the teacher training offered as part of the Indonesia ECED Project and a measure of playgroup quality, key data which were collected in 2013. These additional data from 2013 allow us to predict key elements of sustainability that are not possible if we use the full sample of 432 project playgroups. In Appendix Table 1, we show that there are no significant differences in various baseline center characteristics between the 245 panel project playgroups in the main analytic sample and the full sample of 432 project playgroups. Thus, we consider the main analytic sample to be broadly representative of the 432 project playgroups in these villages.                                                              9 For details of the design of the evaluation see Pradhan et al, 2013. For detailed results on children’s developmental outcomes see Jung and Hasan, 2016 and Brinkman et al. 2017. 10 That is not to say that they had higher total enrollment overall. 5 Summary statistics The main outcome variables of interest in this study are (1) the proportion of project playgroups that remained open in 2016, three years after the end of project funding, and (2) the proportion of project playgroups that remained open in 2016 with two teachers that were trained as part of the project. This latter measure captures a more rigorous definition of sustainability compared to our first measure because it requires the continued employment of teachers who were trained under the project. Of the full sample of 432 project playgroups, 92% were still operating in 2013 at the end of the project and 86% remained open in 2016, see Appendix Figure 1. As noted above, data on teacher training were not available for the full sample of 432 project playgroups so the second measure of sustainability cannot be assessed for this group. For the main analytic sample, the 245 project playgroups with data from all three time points, about 96% were still open in 2013 and by 2016 92% remained in operation (shown as a solid line in Figure 1). Because the project was rolled out in phases and the training was provided in two blocks of 100 hours each, in 2010 our first survey of playgroups showed that only about 29% of the 245 project playgroups reported that they had two trained teachers. However, the proportion of open playgroups with two trained teachers increased between 2013 and 2016 even though the teacher training component of the project ended in 2013 (shown as a dashed line in Figure 1). 11 We argue that open project playgroups with two trained teachers in 2013 and 2016 were those that were particularly successful at attracting and hiring teachers from other project playgroups who had also been trained under the project. This view is supported by the fact that the proportion of open project playgroups with at least one trained teacher declined between 2010 and 2016 (shown as a dotted line in Figure 1). Thus, our second measure of sustainability – the proportion of open project playgroups with two trained teachers – captures those that were most successful in providing quality early childhood education after external funding ended.                                                              11 Since the rollout of the project was not complete, this is understandable. However there is also anecdotal evidence to suggest that there were issues with the fidelity of implementation of the teacher training component which also have contributed to the low share of playgroups with two trained teachers by 2010. 6 Figure 1. Proportion of project playgroups that are open and open with trained teachers in 2010, 2013 and 2016 Note: Plot of proportion of playgroups that (i) remained open, (ii) remained open with at least 1 trained teacher, and (ii) remained open with at least 2 trained teachers, in each year. Former center administrators and teachers provided the date of playgroup closure. Dashed vertical line in 2013 indicates when the Indonesia ECED Project ended. To put these figures in context, consider that of the project playgroups that were open in 2013, 94% of centers remained open in 2016. This sustainability rate is comparable to that of non- project playgroups (98%) including non-project centers (both kindergartens, Islamic kindergartens, and playgroups KB) that were observed in 2013 and remained open in 2016. If we focus specifically on non-project playgroups (KB), 95% that were observed in 2013 remained open in 2016. This suggests that this paper focuses on a pattern that is quite typical in this setting. As noted at the start of this section, the full sample of 432 project playgroups were contacted in 2016 to ascertain whether or not they were still in operation. Sixty of these playgroups had ceased operation by that time and former administrators or teachers were asked to report the key reasons behind the closure. These reasons are summarized in Table 1. Many respondents cited financial reasons for closure such as lack of operational funds (65%) and inability to garner financial support from communities (33%). In most cases, when project playgroups closed the building space was not replaced by other early childhood education programs (70%). In addition, nearly half the closed project playgroups reported that the toys were left in the building (45%) while a third reported that the toys had been lost or broken (33%). Only 22% reported that the toys had been moved to a different early childhood education center. While 40% of teachers were no longer teaching, 26% of teachers were employed in other early childhood education centers and 12% were employed in primary schools. 7 Table 1. Summary statistics of project playgroups that closed Mean (S.D.) * Why did the playgroup close? Lack of operational funds (1=Yes) 0.65 (0.48) Lack of students (1=Yes) 0.37 (0.49) Lack of teachers (1=Yes) 0.35 (0.48) Could not get funding from community (1=Yes) 0.33 (0.48) Too many ECED centers in same neighborhood (1=Yes) 0.08 (0.28) What happened to the building? Building is not used by other ECED/community program (1=Yes) 0.70 (0.46) Building is used by other community program (1=Yes) 0.15 (0.36) Building does not exist anymore (1=Yes) 0.08 (0.28) Building is used by other ECED program (1=Yes) 0.07 (0.25) What happened to the toys? Toys are still left in the building (1=Yes) 0.45 (0.50) Toys no longer exist or are broken (1=Yes) 0.33 (0.48) Toys have moved to other ECED facility (1=Yes) 0.22 (0.42) What happened to the teachers? No longer teaching 0.40 (0.36) Teaching in other ECED center 0.26 (0.36) Teaching in primary/secondary school 0.12 (0.21) Don't know 0.11 (0.22) Moved to other village 0.10 (0.17) * Note: Data from 2016 Indonesia ECED Survey. Reasons for playgroup closure are not mutually exclusive categories (multiple answers possible). N= 60 Qualitative evidence Interviews with former administrators and teachers from closed playgroups were also carried out to provide additional insight to the quantitative data presented in Table 1. Most described a chain of events that led to the closure of their playgroup. As one interviewee said, “When the project ended, we no longer had funds to pay the teachers and replace the books and toys that were damaged. So the teachers quit. And the students left. Nobody stepped up to keep the center.” Many of the project playgroups that eventually closed had tried to remain open for some time, for example, one former teacher described, “After the project ended, the center continued to operate for a year. During this time, the center relied on funding from the village government. The center held a meeting with local parents, requesting them to pay tuition fees in order to maintain the center. But this was not successful.” Indeed, our quantitative data suggest that a strategy employed by many playgroups to sustain their programs was to charge fees. As shown in Appendix Table 2, only 32% of playgroups that would eventually close were charging fees in 2010. In contrast, 51% of playgroups that would remain open were charging fees in 2010. The proportion of playgroups charging fees rapidly expanded between 2010 and 2013, for both those that would eventually close (59%) and those that would remain open (70%). Moreover, project playgroups 8 that would remain open were charging higher median fees relative to those that would eventually close in both 2010 and 2013.12 In other settings, disagreement between different stakeholders contributed to the closure of project playgroups. “There was a conflict between the board members of the village education department and the village chief. The chief did not trust the board members and decided that the center would only receive support from the village government for one year, not indefinitely.” Thus, the end of the project presented considerable financial challenges to the playgroups. While many found ways to continue providing early childhood education, some struggled to find both the financial and human resources needed to sustain their programs. This situation was exacerbated by the fact that some teachers under the project received monthly honoraria (250,000 rupiah) that far exceeded typical wages in the villages for preschool teachers (50,000 rupiah). Others struggled to receive any regular compensation for their work at all. IV. Methodology We examine five categories of predictors of sustainability for our main analytic sample of 245 project playgroups, which are summarized in Table 2 below.13 These predictor variables were measured during the project period in 2010 and/or in 2013. The first category of predictors is finance, measured using four variables: (i) amount of funding from the government, (ii) amount of funding from non-governmental sources, (iii) amount of funding from parents, and (iv) share of annual expenditure for teacher salaries.14                                                              12 Median fees ranged from IDR 5,000 per month to IDR 10,000 per month. Rural households at the time reported a monthly wage of IDR 1.7 million as per Bureau of Labor Statistics (2016). 13 See Appendix Figures 2 to 4 for plots of distributions of some of these variables. 14 See Appendix Figure 5 for plots of distributions of teacher salaries. 9 Table 2. Summary statistics of project playgroup characteristics Year of survey: 2010 2013 Diff Mean Mean Mean (S.D.) (S.D.) (S.E.) 1. Finance (units: million IDR) Amount from government in 2009-2013 . 22.25 . . (49.53) . Amount from non-government in 2009-2013 . 0.60 . . (3.44) . Amount from parents in 2009-2013 . 4.70 . . (19.41) . Proportion of annual expenditure for teacher salaries 0.15 0.18 0.03*** (0.07) (0.08) (0.01) 2. Quality Score from classroom observation (ranges from 1-7) . 3.09 . . (0.98) . Proportion of teachers with post-secondary degree 0.20 0.23 0.03 (0.34) (0.31) (0.02) Proportion of teachers with teaching experience 0.31 0.36 0.06* (0.42) (0.45) (0.03) 3. Market condition (unit: per 100 children in village) Number of kindergartens 0.71 0.50 -0.21** (1.45) (0.56) (0.10) Number of other playgroups 0.35 0.28 -0.08 (1.45) (0.54) (0.10) Distance to village center (km) 1.38 . (3.46) . Distance to nearest playgroup (km) 2.14 . (3.88) . 4. Supplementary services Provides weekly food program (1=Yes) 0.44 0.26 -0.18*** (0.48) (0.44) (0.03) Provides weekly vitamin (1=Yes) 0.08 0.07 -0.01 (0.25) (0.26) (0.02) Provides anti-worming medication (1=Yes) 0.46 0.63 0.17*** (0.49) (0.48) (0.04) 5. Community support Parental involvement index (z-score) 0.00 0.13 0.13 (1.02) (1.07) (0.08) * p<0.10, ** p<0.05, *** p<0.01 Note: Data from Indonesia ECED Surveys. Sample size is 245 project playgroups in each year. Cells with . indicate that the variable was not collected that year. Difference column is the difference between 2013 and 2010. Robust standard errors clustered by center. Classroom observation is measured using the ECERS-R. Parental involvement is constructed from a principal component analysis of four items: parents are involved in activities with teachers and students in the center; parents are involved in cleaning the center; parents are involved in supplementary food program in the playgroup; parents are involved in acquiring toys and materials in the playgroup. The index is normalized to have mean 0 and standard deviation 1 in 2010. A higher value of the index means more parental involvement. 10 To construct the funding amount variables, we use data on the total amount of funding during the project period (2009-2013), which was collected only in the 2013 survey. For the expenditure variable, we use annual expenditure data on teacher salaries, which were collected separately in 2010 and 2013. Table 2 shows that during the project period, the largest financial support (beyond the approximately 77 million rupiah project funds per center) came from government sources (22.25 million rupiah). Project playgroups received modest financial support from parents (4.70 million rupiah) and little support from non-governmental sources (0.60 million rupiah) during the project period. In addition, the average project playgroup significantly increased the share of annual expenditure on teacher salaries from 15% in 2010 to 18% in 2013.15 Our second predictor category is playgroup quality, which is captured using three variables: (i) observed quality in the playgroup, (ii) proportion of teachers with a post-secondary degree, and (iii) proportion of teachers with teaching experience in early childhood education. Our measure of observed quality was collected only in 2013 and used the Early Childhood Environment Rating Scale – Revised (ECERS-R) (Harms, Clifford and Cryer 2005). The ECERS- R uses classroom observations, playgroups in this case, to assess the various interactions that occur between and among teachers, students, and parents. Each playgroup was assessed by two raters on a 7-point Likert scale, ranging from 1 = inadequate, 3 = minimal, 5 = good, to 7 = excellent. In an effort to align the ECERS-R data with the reality of the Indonesian context, we compared ECERS-R to Indonesia’s national standard and found that 28 out of 43 ECERS-R items were discussed in the national standard (see Table 4 in Brinkman et al. 2017). We, therefore, calculate playgroup quality using only the 28 items that were found to be common between ECERS-R and the national standard. The average playgroup in our analytic sample has a score of 3.09 – which corresponds to minimal quality on the ECERS-R scale. 16 We also measured playgroup quality using teacher characteristics. On average, the proportion of teachers with post- secondary education marginally increased from 20% in 2010 to 23% in 2013, and the proportion of teachers with teaching experience in early childhood education significantly increased from 31% in 2010 to 36% in 2013. The third category is market condition, which we measure using four variables: (i) number of kindergartens per 100 children in the village, (ii) number of other playgroups per 100 children in the village, (iii) distance to the village center, and (iv) distance to the nearest neighboring playgroup. The first and second variables capture the extent to which the project playgroups are faced with competition from complementary services (kindergartens) and substitute services (other playgroups). While several types of early childhood services exist in Indonesia, the two most common types of pre-primary education are playgroups and kindergartens. Playgroups cater to children ages 3-4 with a focus on play-based learning while kindergartens are intended for children ages 5-6 with a focus on academic preparation for primary school. Thus, for the project playgroups, other playgroups operating in the same village but not funded by the project can be seen as substitutes while kindergartens operating in the same village function as complements. Table 2 shows that, on average, there were more kindergartens than playgroups. In 2010, there were 0.71                                                              15 Appendix Table 3 reports these numbers separately for open and closed playgroups between 2010 and 2013. Appendix Table 4 examines this information only for open centers for which the data are available from 2013 to 2016. Appendix Table 5 reports teacher salaries. 16 While we also collect data on student outcomes at the center level using the Early Development Instrument (EDI) (Janus and Offord 2007), which measures children’s school readiness and focuses on five domains: physical health and well-being, social competence, emotional maturity, language and cognitive skills, and communication and general skills We do not apply these as a measure of playgroup quality as it could reflect sorting on the part of children and their families. 11 kindergartens and 0.35 playgroups per 100 children; in 2013, these figures declined to 0.50 kindergartens and 0.28 playgroups per 100 children. Project playgroups were on average 1.38 km away from the center of the village and spaced 2.14 km away from the nearest neighboring early childhood education center. The fourth category is supplementary services, which are measured using three variables: (i) weekly provision of food programs, (ii) weekly provision of vitamin supplements, and (iii) provision of anti-worming medication. In poor, rural villages in Indonesia, playgroups can offer a space not only for early education but also for supporting other health and nutrition programs for toddlers. In 2010, centers offered weekly food programs (44%), weekly vitamin supplements (8%), and anti-worming medication (46%). In 2013, a significantly smaller share of centers offered weekly food programs (26%) and a significantly larger share of centers provided anti-worming medication (63%). The last category of predictors is community support, which we measure using parental involvement in the center. Given that the Indonesia ECED project was implemented under a community-based approach, volunteer efforts from parents and other community members are likely to be important for the sustainability of the playgroups established under the project. The parental involvement index is constructed using a principal component analysis of four items: parents are involved in activities with teachers and students in the playgroup; parents are involved in cleaning the playgroup; parents are involved in providing food in the playgroup; and parents are involved in acquiring toys and materials in the playgroup. We normalize the index to have mean 0 and standard deviation 1 in 2010. A higher value of the index means more parental involvement. Between 2010 and 2013, parental involvement in the average playgroup increased by 0.13 standard deviation, which is equivalent to a 10% increase in parents helping teachers with educational activities in the playgroups and a 7% increase in parents helping with cleaning and maintenance of the playgroups. This increase in parental involvement is striking given that in other playgroups in the same villages (i.e., non-project playgroups), parental involvement decreased by 0.09 standard deviations between 2010 and 2013. Thus, these five categories – finance, quality, market condition, supplementary services, and community support – represent key constructs that are theoretically important for sustainability. Next, we empirically examine the extent to which these sets of predictors are important for sustainability of project playgroups in our study context. Empirical strategy The aim of our analysis is to explore which variables predict sustainability. We run the following logistic regression model: (1) E 1|X α , , where E 1| is the population mean that the th center was open at 2016, 1. ∙ is the logistic cumulative distribution function (CDF). The constant term is α and is a column of k coefficients in the category of interest. We re-run the equation for our second measure of sustainability, where E 1| is the population mean that the th center was open with two trained teachers at 2016 1 . In both model specifications, positive coefficients imply better outcomes (i.e., more sustainability) for the playgroups. Robust standard errors clustered at the playgroup-level are used throughout. For predictor variables that were collected at both 2010 and 2013, our models include the variable at 2010 and the change in the variable between 2010 and 2013 when data 12 for both years are available. This allows us to examine if trends in the predictor variables during the project period are associated with the outcome after the project ended. For example, the analysis for finance has five coefficients: (i) amount of funding from the government during the project period, (ii) amount of funding from non-governmental organizations during the project period, (iii) amount of funding from parents during the project period, (iv) share of annual expenditure for teacher salaries in 2010, and (v) change in the share of annual expenditure for teacher salaries between 2010 and 2013. For each outcome measure, we perform five logistic regressions (one for each category of predictors). Within each category of predictors, we are interested in the variables that are most predictive of sustainability (controlling for the other variables in the same category). In the results below, we present coefficient estimates as average marginal effects (instead of logit or odds ratio). Our average marginal effect is the average of the marginal effects over all the possible values of independent variables. With binary or categorical predictors, marginal effects measure the change in predicted probability for a one-unit change in the predictor. With continuous predictors, marginal effects measure the instantaneous rates of change. When the relationship between the outcome and predictor is non-linear, the marginal effects for continuous predictors do not approximate to the change in outcome associated with a one-unit change in the predictor. For ease of interpretation, we also plot the marginal effects for continuous predictors that are statistically significant. Finally, we examine the relationship between sustainability and the five categories of predictors jointly. We use a two-step process. In the first step, we estimate the predicted probability (p-hat) from the regression in equation 1. This results in five predicted probabilities (p-hat) since we estimate one from each of the five categories. Then, we adjust these predicted probabilities by dividing each by its standard deviation. This adjustment ensures that each p-hat is treated on the same scale. The results are standardized indices of the predictors, where the weights have been chosen such as to maximize the predictive power of the index. In the second step, we use the five adjusted p-hat to jointly predict sustainability. We perform the same logistic regression as equation 1 except is now a column of k=5 coefficients of the adjusted p-hats. We are interested in the relative magnitude of these coefficients because they indicate the degree to which each predictor category predicts sustainability. This two-step process is performed for each of our two outcomes of sustainability: being open in 2016 and being open with two trained teachers in 2016. V. Results The results of regressing sustainability on finance characteristics are presented in Table 3. Project playgroups with a higher proportion of annual expenditure on teacher salaries during the project period were significantly more likely to be open in 2016. Moreover, playgroups that increased the share of annual expenditure on teacher salaries over the course of the project period were significantly more likely to remain open (column 1) and remain open with two trained teachers (column 2).17                                                              17 In alternative specification we reformulate this to include the total amount of funding from all sources and include dummies for whether or not funding was received from government, non-government or parents. The results are very similar and available upon request. 13 Table 3. Logistic regression of project playgroup sustainability on finance characteristics (1) (2) Playgroup is open in Playgroup is open in 2016 with 2016 (1=Yes) two trained teachers (1 = Yes) Finance (units: 100 million IDR) Amount from government in 2009-2013 0.28 0.03 (0.28) (0.06) Amount from non-government in 2009-2013 0.70 0.08 (0.48) (0.92) Amount from parents in 2009-2013 0.33 0.04 (0.35) (0.14) Proportion of annual expenditure on teacher 0.48* -0.18 salaries in 2010 (0.28) (0.52) Change in proportion of annual expenditure 0.40* 0.73* on teacher salaries from 2010-2013 (0.21) (0.38) Log-likelihood -62.63 -154.29 Observations 245 245 * p<0.10, ** p<0.05, *** p<0.01 Note: Robust standard errors clustered by center. Coefficients are marginal effects. Figure 2 shows the relationship between the playgroup remaining open and the change in proportion of annual expenditure on teacher salaries. The predicted probability that a playgroup remained open and open with two trained teachers are presented as solid and dashed lines, respectively. The predicted probability of a playgroup remaining open and open with two trained teachers (dashed line) is 90% and 32%, respectively for a center that made no change in teacher expenditures during the project period. Playgroups that increased their annual expenditure on teacher salaries by 50% between 2010 and 2013 is estimated to have a 98% probability of remaining open and a 70% chance of being open with two trained teachers in 2016.18 These results suggest the importance of financial commitment to teachers for project sustainability. It is noteworthy that the amounts of funding from various sources are positively associated with sustainability – yet these estimates are imprecise and indistinguishable from zero (controlling for other variables in the model).                                                              18 See appendix figure 8 for further details on how playgroups chose to pay teacher salaries. 14 Figure 2. Probability of sustainability and change in expenditure on teacher salaries Note: Plot of predicted probability that a playgroup remained open (solid line) and remained open with 2 trained teachers (dashed line). Variable on x-axis was measured during the project period (2010 and 2013) and variable on y- axis was measured after the project period (2016). The model holds constant the following variables at its mean level: amount of funding from the government during the project period, amount of funding from non-governmental organizations during the project period, amount of funding from parents during the project period, and share of annual expenditure for teacher salaries in 2010. Table 4 presents the results of regressing sustainability on quality characteristics. While teacher characteristics based on education level and teaching experience are not significant predictors of sustainability (controlling for other variables), observed quality is a significant predictor. A one-unit increase in the ECERS-R scale is associated with a 3 percentage point increase in the predicted probability of a project playgroup remaining open (column 1) and a 4 percentage point increase in the predicted probability of a project playgroup remaining open with two trained teachers (column 2). An increase of 3 percentage points in the predicted probability of remaining open (statistically significant at p<0.1) is modest, given that 92% of centers remained open in 2016. Table 4. Logistic regression of project playgroup sustainability on quality characteristics (1) (2) Playgroup is open in Playgroup is open & two 2016 (1=Yes) trained teachers (1 = Yes) Quality Observed quality in 2013 0.03* 0.04 (0.02) (0.03) Prop. of teachers with post-secondary degree 0.08 0.05 in 2010 (0.07) (0.12) Change in prop. of teachers with post-secondary 0.00 -0.06 degree from 2010-2013 (0.07) (0.09) Prop. of teachers with teaching experience 0.04 -0.05 in 2010 (0.06) (0.09) 15 Table 4. Logistic regression of project playgroup sustainability on quality characteristics (1) (2) Playgroup is open in Playgroup is open & two 2016 (1=Yes) trained teachers (1 = Yes) Change in prop. of teachers with teaching 0.00 0.06 experience from 2010-2013 (0.04) (0.08) Log-likelihood -65.97 -156.86 Observations 245 245 * p<0.10, ** p<0.05, *** p<0.01 Note: Robust standard errors clustered by playgroup. Coefficients are average marginal effects. Figure 3 plots the relationship between observed quality and sustainability of project playgroups. The mean quality score in our sample was 3.09 (see Table 2).19 A project playgroup that received a quality rating of minimal (quality score = 3) during the project period has a predicted probability of 92% that it will remain open and a predicted probability of 34% that it will remain open with two trained teachers after the project ends (holding all other variables at their mean). The rate of sustainability increases as playgroup quality increases. A project playgroup that received a rating of good (quality score = 5) during the project period has a predicted probability of 97% that it will remain open and a predicted probability of 42% that it will remain open with two trained teachers. Figure 3. Probability of sustainability and observed playgroup quality Note: Plot of predicted probability that a playgroup remained open (solid line) and remained open with 2 trained teachers (dashed line). Variable on x-axis was measured during the project period (2010 and 2013) and variable on y-axis was measured after the project period (2016). The model holds constant the following variables at its mean level: proportion of teacher with post-secondary education in 2010, change in proportion of teacher with post- secondary education between 2010-2013, proportion of teachers with teaching experience in 2010, change in proportion of teachers with teaching experience between 2010-2013.                                                              19 Recall that we apply the ECERS-R methodology but only focus on the items that are common between the national standard and the ECERS-R. We refer to this as the quality score. 16 These results suggest that classroom observations that measure how teachers, students, and parents are interacting between and among each other (namely the ECERS-R) are capturing elements of playgroup quality beyond what can be predicted from student’s school readiness, teachers’ education and teaching experiences alone. Thus, observed playgroup quality is a strong predictor of whether project playgroups are sustained after the initial funding period.20 Table 5 summarizes the relationship between sustainability and market conditions. First, we do not find support for the fact that distance to the village center or distance to neighboring playgroup services mattered in our setting. Second, project playgroups located in villages with a higher density of kindergartens during the project period were significantly more likely to be open (column 1) and remain open with two trained teachers (column 2) – holding constant all other variables in the model. As shown in Figure 4, below, the predicted probability that a project playgroup remains open increases from 82% to 97% as the number of kindergartens per 100 children increases from 0 to 1. Similarly, the predicted probability that a project playgroup is open with two trained teachers jumps from 29% to 41% as the number of kindergartens per 100 children increases from 0 to 1. We view kindergartens as services that complement playgroups given the age of students intended for each. Therefore, in villages where both kindergartens and playgroups are sought after, we would expect a positive relationship between the density of kindergartens and sustainability of project playgroups. Our results are consistent with this hypothesis.21 Table 5. Logistic regression of project playgroup sustainability on market conditions (1) (2) Playgroup is open Playgroup is open in 2016 with in 2016 (1=Yes) two trained teachers (1 = Yes) Market conditions Number of kindergartens per 100 children in 2010 0.15** 0.12* (0.07) (0.06) Change in number of kindergartens per 100 children -0.09 -0.11 from 2010-2013 (0.06) (0.07) Number of other playgroups per 100 children in 2010 -0.11 -0.10 (0.07) (0.08) Change in number of other playgroups per 100 children 0.10 0.08 from 2010-2013 (0.06) (0.07) Distance to village center (km) 0.00 0.01 (0.03) (0.02) Distance to nearest neighboring playgroup (km) 0.04 -0.01 (0.03) (0.01) Log-likelihood -63.83 -155.81 Observations 245 245 * p<0.10, ** p<0.05, *** p<0.01 Note: Robust standard errors clustered by center. Coefficients are average marginal effects.                                                              20 This finding is in close agreement to the findings in the broader literature on preschool center quality. See for instance, Brinkman et al, 2017. 21 See Appendix Figure 6. 17 Figure 4. Probability of sustainability and number of kindergartens in the village Note: Plot of predicted probability that a playgroup remained open (solid line) and remained open with 2 trained teachers (dashed line). Variable on x-axis was measured during the project period (2010 and 2013) and variable on y- axis was measured after the project period (2016). The model holds constant the following variables at its mean level: change in number of kindergartens per 100 children from 2010-2013, number of other playgroups per 100 children in 2013, change in number of other playgroups per 100 children from 2010-2013, distance to village center and distance to nearest neighboring playgroup. In contrast to kindergartens, we view other playgroups as services that directly compete with or substitute for project playgroups. Consistent with this view, Table 5 shows that the density of other playgroups is negatively associated with sustainability (although the coefficient is not significant in either column). However, results in column 1 also show a significant positive coefficient for the change in density of other playgroups. It is somewhat surprising that project playgroups are more likely to remain open in villages where the number of competing playgroups is increasing. However, this is likely to be capturing the increase in overall demand for playgroups in general during the project period. In other words, villages where playgroups are generally sought after are likely to witness both an increase in the number of playgroups available and an increase in the probability of a project playgroup remaining open. Between column 1 and column 2, the coefficient for the change in density of other playgroups declines and becomes indistinguishable from zero. This suggests that while overall demand for playgroups may be increasing (and therefore suggesting that a project playgroup remains open), project playgroups must compete with other existing playgroups to hire teachers that were trained under the project.22 Next, we turn to the relationship between project playgroup sustainability and provision of supplementary services in Table 6. The predicted probability that a project playgroup remains open with two trained teachers is 14% (statistically significant at p<0.1) greater for those that provided weekly food programs in 2010 than for those that did not provide such services in 2010 (holding all else constant). However, project playgroups that changed their provision of food programs between 2010 and 2013 – going from not providing to providing – are 11% less likely to be open (statistically significant at p<0.05) in 2016 and 19% less likely to be open (statistically significant at p<0.01) with two trained teachers in 2016. Figure 5 also shows this relationship.                                                              22 Appendix Figure 6 plots typical distances to village center for open and closed project playgroups. 18 Table 6. Logistic regression of playgroup sustainability on supplementary services (1) (2) Playgroup is open in Playgroup is open 2016 with two trained in 2016 (1=Yes) teachers (1 = Yes) Provided food program in 2010 (1=Yes) 0.08 0.14* (0.05) (0.08) Change in provision of food program from 2010-2013 -0.11** -0.19*** (0.04) (0.06) Provided vitamin in 2010 (1=Yes) -0.09 -0.04 (0.07) (0.17) Change in provision of vitamin from 2010-2013 0.10 0.12 (0.07) (0.13) Provided anti-worming medication in 2010 (1=Yes) 0.00 0.04 (0.04) (0.08) Change in provision of anti-worming medication 0.02 0.05 from 2010-2013 (0.04) (0.06) Log-likelihood -63.75 -152.46 Observations 245 245 * p<0.10, ** p<0.05, *** p<0.01 Note: Robust standard errors clustered by playgroup. Coefficients are average marginal effects. Figure 5. Probability of sustainability and change in provision of food program Note: Plot of predicted probability that a playgroup remained open (solid line) and remained open with 2 trained teachers (dashed line). Variable on x-axis was measured during the project period (2010 and 2013) and variable on y- axis was measured after the project period (2016). The model holds constant the following variables at its mean level: provision of food program in 2010, provision of vitamin supplements in 2010, provision of anti-worming medication in 2010, change in provision of vitamin supplements from 2010-2013, and change in provision of anti-worming medication from 2010-2013. 19 This negative relationship between change in provision of a food program and sustainability is likely to be driven by the strategic reallocation of expenditures during the project period among the project playgroups that were sustained. Recall that in Table 3, we found that increasing the share of expenditures on teacher salaries is positively associated with sustainability. Our results suggest that among the project playgroups that were sustained, much of the reallocation of expenditures toward teacher salaries came from cutting back on weekly provision of food program. Finally, we explore the relationship between sustainability and community support. We find that centers with higher levels of parental involvement during the project period were significantly more likely to remain open (column 1, statistically significant at p<0.1) and remain open with two trained teachers (column 2, though these results were statistically insignificant) – holding constant the change in parental involvement during the project period although the estimates are modest. Figure 6 shows that the predicted probability that a project playgroup remains open in 2016 is 91% for those with the average level of parental involvement. A one standard deviation increase in parental involvement is associated with a predicted probability of 95% that a project playgroup remains open in 2016. Table 7. Logistic regression of project playgroup sustainability on community support (1) (2) Playgroup is Playgroup is open in 2016 open in 2016 with two trained teachers (1=Yes) (1 = Yes) Community support Parental involvement (z-score) in 2010 0.04* 0.03 (0.02) (0.04) Change in parental involvement from 2010-2013 -0.03 0.03 (0.02) (0.03) Log-likelihood -67.93 -155.87 Observations 245 245 * p<0.10, ** p<0.05, *** p<0.01 Note: Robust standard errors clustered by center. Coefficients are average marginal effects. Figure 6. Probability of sustainability and parental involvement 20 Note: Plot of predicted probability that a playgroup remained open (solid line). Variable on x-axis was measured during the project period (2010 and 2013) and variable on y-axis was measured after the project period (2016). The model holds constant the following variables at its mean level: change in parental involvement from 2010-2013. Next, we examine the relationship between sustainability and the five categories of predictors jointly. Figure 7 shows the predicted probabilities of sustainability estimated from each of the categories of predictors. As shown, the variation of the predicted probabilities (p-hat) differs across each category so we adjust them by dividing by its standard deviation before they are entered into the regression. This adjustment ensures that each p-hat is treated on the same scale. As discussed in the empirical strategy, we use these five adjusted p-hat to jointly predict sustainability. Figure 7. Histograms of observed status and probabilities of sustainability Panel A. Center is open in 2016 A. Finance B. Quality C. Market condition 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 Density Density Density 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Prob. of open in 2016 Prob. of open in 2016 Prob. of open in 2016 Predicted: mean=0.918 sd=0.061 Predicted: mean=0.918 sd=0.045 Predicted: mean=0.918 sd=0.082 D. Supplementary services E. Community support 0 20 40 60 80 100 0 20 40 60 80 100 Density Density 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Prob. of open in 2016 Prob. of open in 2016 Predicted: mean=0.918 sd=0.064 Predicted: mean=0.918 sd=0.028 Predicted Observed Observed: mean=0.918 sd=0.274 21 Panel B. Center is open with two trained teachers in 2016 A. Finance B. Quality C. Market condition 80 80 80 20 40 60 20 40 60 20 40 60 Density Density Density 0 0 0 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Prob. open w/ 2 trained teachers in 2016 Prob. open w/ 2 trained teachers in 2016 Prob. open w/ 2 trained teachers in 2016 Predicted: mean=0.347 sd=0.084 Predicted: mean=0.347 sd=0.049 Predicted: mean=0.347 sd=0.081 D. Supplementary services E. Community support 80 80 20 40 60 20 40 60 Density Density 0 0 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Prob. open w/ 2 trained teachers in 2016 Prob. open w/ 2 trained teachers in 2016 Predicted: mean=0.347 sd=0.102 Predicted: mean=0.347 sd=0.066 Predicted Observed Observed: mean=0.347 sd=0.477 Notes: Panel A shows probability of open in 2016 and panel B shows probability of open with two trained teachers in 2016. Pink color indicates predicted probability (p-hat) and green color indicates observed values. The results of the joint estimation are shown in Table 8 below. For both measures of sustainability, three categories are particularly informative and significant at the 5% level: finance, market condition, and supplementary services. In particular, local market conditions are the strongest predictor category of whether a playgroup is open in 2016. For whether a playgroup is open with two trained teachers in 2016, the provision of supplementary services is the strongest predictor category. Table 8. Logistic regression of project playgroup sustainability on five categories (1) (2) Playgroup is open in Playgroup is open in 2016 with 2016 (1=Yes) two trained teachers (1 = Yes) Adjusted p-hat from finance variables 0.03*** 0.07** (0.01) (0.03) Adjusted p-hat from quality variables 0.03* 0.03 (0.02) (0.03) Adjusted p-hat from market condition variables 0.04*** 0.06** (0.01) (0.03) Adjusted p-hat from supplementary services variables 0.02** 0.08*** (0.01) (0.03) Adjusted p-hat from community support variables 0.02 0.04 (0.02) (0.03) Observations 245 245 * p<0.10, ** p<0.05, *** p<0.01    Note: Robust standard errors clustered by center. Coefficients are average marginal effects. For each category, adjusted p- hat is the predicted probability (estimated from the regressions presented in Tables 3-7) divided by its standard deviation.   22 VI. Discussion and Conclusion The analyses in this paper highlight several key results. We summarize each of these below and consider their implications for project design and implementation. First, the vast majority (86%) of the 432 playgroups established under the project remained open three years after development aid ended. This is particularly high given evidence from comparable settings. For instance, evidence from 657 community-driven development micro- infrastructure projects built between 1999 and 2007 in Indonesia suggest that only 72% were fully functioning when revisited between 2006 and 2008 (Kulongoski 2009). In particular, project playgroups coped with the cessation of donor funding by introducing and in some cases increasing student fees. Both our survey data and qualitative interview data show that project playgroups that closed had struggled to find the financial and human resources needed to continue operation. The fact that many project playgroups would choose not to charge fees at the outset was not expected at the project design stage. There was no explicit instruction for or against this in the project appraisal document. Thus one implication that follows is that project design needs to consider local ability and willingness to pay when designing such interventions. One possibility would have been to allow project playgroups to pick their own fee levels and to build in a mechanism under the project whereby eligible beneficiary households receive direct subsidies to pay these fees. Second, we show that project playgroups that increased their share of expenditures for teacher salaries during the project period were significantly more likely to be sustained after the project ended. We find evidence that project playgroups reallocated their resources towards teacher salaries by cutting down on some supplementary services (such as weekly food programs) and scaling up others (such as deworming). Motivated teachers are critical for delivering high-quality education. In order to ensure that projects such as this attract the most qualified talent, it is important not only that salaries be provided but that they do not distort local market conditions. As noted earlier, some teachers under the project received payments that far exceeded local norms. Others struggled to receive any compensation at all. Neither situation is desirable and can be avoided if project design stipulates mechanisms (if not levels) of compensation. Third, the quality of project playgroups – as measured by classroom observations – is a key predictor of sustainability. We find that direct observations of the playgroups are able to capture elements of the centers that are predictive of sustainability, beyond what can be predicted from, student’s school readiness, teachers’ education and teaching experiences alone. Given that these are community-based services and that parents are often able to directly observe quality – both the state of physical infrastructure as well as the quality of teacher and child interactions – it is imperative that projects build in mechanisms for tracking and improving quality throughout the project. We also find that market conditions, as measured by the number of kindergartens per 100 children, and community support, as measured by the degree of parental involvement in project playgroups are both strong predictors of sustainability. Both variables indicate community-wide support for investing in early childhood education. As such, market conditions need to be studied and efforts made to adapt to these at the design stage. This will bear not only on what types of services are established but how much these services charge and how much they remunerate their teachers. Finally, when we jointly examine the categories of predictors (finance, quality, market conditions, supplementary services, and community support), local market condition is the strongest predictor category of a center remaining open three years after the project ended. For 23 higher degrees of sustainability, the provision of supplementary services is the strongest predictor category. In interpreting these results it bears noting that sustainability, while important in its own right would be rendered unappealing if their continued operation meant excluding the poorest children. Appendix Table 6 shows the average household wealth of children enrolled in project playgroups in 2013. The household wealth of children who attended project playgroups that remained open in 2016 and that of children who attended those that eventually closed in 2016 are indistinguishable from each other. The same is true at the village level. Villages where all project playgroups stayed open are no richer or poorer on average than those villages where some of the project playgroups had closed.23 Taken together, this result suggests that at least in the present setting, sustainability of project playgroups did not lead to exclusion from them. We note that this is the first study that examines multiple factors related to sustainability at the same time. To our knowledge the literature typically examines each of these hypotheses separately and rarely does a paper examine more than one of these hypotheses together. That said, there are limitations in this paper that warrant noting. Some of our estimates have the expected signs and magnitudes but our sample is not large enough for these estimates to achieve usual levels of significance even after running separate regressions for the different predictors to avoid inefficient estimates driven by multicollinearity. Due to this lack of power, some of the estimates are significant only at p<0.1. Having tested some of the most commonly used hypotheses in the literature – ranging from those found in studies of hospital sustainability in the United States (Succi et al., 1997) to those found in studies of the sustainability of a reading program in Zambia (Meki Kombe and Harmon, 2017) – we have reasonably high confidence in the validity of the model though the usual disclaimers regarding external validity apply. Beyond the empirical patterns documented in this paper, the process of having documented these patterns leads us to several additional observations. These touch on the sustainability and design of international development projects, particularly those in early childhood education. Strategies to ensure sustainability of project activities should be introduced before the end of the project, ideally during the early stages of project design and implementation. These could include training for communities on how to go about securing financial support from diverse sources and on prioritizing spending on critical inputs such as teacher salaries. Community awareness raising campaigns should be run for the life of the project – not only at the start. This will allow initiatives to take on board newly eligible beneficiaries, for instance new parents. Another key strategy is to directly consult with important stakeholders, who, more often than not, are the parents in early childhood education projects. These consultations will clarify the local demand for early childhood education, including the possibility of charging student fees. Engaging with other key local stakeholders such as the village health worker will help identify the need to provide additional early childhood services (i.e., supplementary food programs). With a growing focus on the importance of supporting women’s participation in the labor force, it will be imperative to consider what options for work exist in these communities. This will also help project designers to assess whether the proposed services are offered at the right time and for the right duration to be of use in freeing up time for women, in particular, and households, in general. This will also help develop strong local ownership of the early childhood education center. Moreover, future international development projects in early childhood education will greatly benefit from conducting a careful assessment of what types of early childhood programs already exist in the local community, how these various programs are utilized by families, and the                                                              23 See Appendix Table 7. 24 demand and supply of each type of service in order to better understand the market conditions before project implementation. Doing so can help ensure that donor-funded interventions have a higher likelihood of survival after project closure. 25 References Bamberger, M. & Cheema S. (1990). Case Studies of Project Sustainabiltiy: Implications for Policy and Operations from Asian Experience. The World Bank: Washington, DC. Barnett, W. Steven. (1997). Costs and financing of early child development programs. In Early child development: Investing in our children’s future, ed. Mary E. Young. Amsterdam: Elsevier. Brinkman, S. A., Hasan, A., Jung, H., Kinnell, A., Nakajima, N., & Pradhan, M. (2017). The role of preschool quality in promoting child development: evidence from rural Indonesia. European Early Childhood Education Research Journal, 1-23. Bryk, A. S. Sebring, P.B., Allensworth, E., Luppescu, S., & Easton, J.Q. (2010). Organizing schools for improvement: Lessons from Chicago. Chicago, IL: University of Chicago Press. Chapman, D. W., & Moore, A. S. (2010). A meta-look at meta-studies of the effectiveness of development assistance to education. International Review of Education, 56(5/6), 547-565. Evans, Judith L., Robert G. Meyers, and Ellen Ilfeld. (2000). Early childhood counts: A programming guide on early childhood care for development. Washington, DC: World Bank Publications. Bureau of Labor Statistics, Government of Indonesia. (2016). Labor Force Situations in Indonesia. February 2016. Harms, T., Clifford, R. M., and Cryer, D. (2005). Early Childhood Environment Rating Scale - Revised Edition. New York: Teachers College Press. Hasan, A., Hyson, M., & Chang, M. (Eds.). (2013). Early Childhood Education and Development in Poor Villages of Indonesia: Strong Foundations, Later Success. Washington DC: World Bank. Janus, M. and Offord, D. (2007). Development and psychometric properties of the Early Development Instrument (EDI): a measure of children’s school readiness. Canadian Journal of Behavioural Science 39 (1): 1–22. Jung, H., & Hasan, A. (2016). The impact of early childhood education on early achievement gaps in Indonesia. Journal of Development Effectiveness, 8(2), 216-233. Kershaw, P., Forer, B., & Goelman, H. (2005). Hidden fragility: Closure among licensed child- care services in British Columbia. Early Childhood Research Quarterly, 20(4), 417-432. Lam, Marcus, Klein, Sacha, Freisthler, Bridget & Weiss, Robert, E. (2013). Child center closures: Does nonprofit status provide a comparative advantage? Child and Youth Services Review, 36(3): 525-534. Lockwood, H., A. Bakalian, and W. Wakeman. (2003). Assessing Sustainability in Rural Water Supply: The Role of Follow-up Support to Communities: Literature Review and Desk Review of Rural Water Supply and Sanitation Project Documents. Washington, DC: World Bank. Kulongoski, T. (2009). Operation and Maintenance Review for Program Pengembangan Kecamatan (PPK) and Program Nasional Pemberdayaan Masyarakat (PNPM). Meki Kombe, C. L., & Herman, C. (2017). Can education innovations be sustained after the end of donor funding? The case of a reading intervention programme in Zambia. Educational Review, 1-21. OECD (2017). Net ODA (indicator). doi: 10.1787/33346549-en 26 RESULTS (2017). Investing in Every Child’s Early Years: World Bank Contributions. Available online at: https://results.org/blog/investing_in_every_childs_early_years_world_bank_contributions/. Accessed on August 12, 2019. Sayre, R. K., Devercelli, A. E., Neuman, M. J., & Wodon, Q. (2015). Investing in early childhood development: review of the World Bank's recent experience. World Bank Publications. Scheirer, M. A., & Dearing, J. W. (2011). An agenda for research on the sustainability of public health programs. American Journal of Public Health, 101(11), 2059-2067. Shediac-Rizkallah, M. C., & Bone, L. R. (1998). Planning for the sustainability of community- based health programs: conceptual frameworks and future directions for research, practice and policy. Health Education Research: Theory & Practice, 31(1): 87-108. Succi, Melissa J, Lee, Shoou-Yih D, & Alexander, Jeffrey A. (1997). Effects of market position and competition on rural hospital closures. Health Services Research, 31(6): 679-699. UNESCO (2015). “Education for All Global Monitoring Report Statistical Tables” (2015). Available online http://en.unesco.org/gem-report/node/6 United Nations (2015). Transforming our world: the 2030 Agenda for Sustainable Development. Available online at: http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E World Bank (2014). Indonesia - Early Childhood Education and Development Project (ECED) Implementation Completion and Results Report. Washington, DC; World Bank Group. Available online at: http://documents.worldbank.org/curated/en/854891468267037215/Indonesia-Early- Childhood-Education-and-Development-Project-ECED 27 Appendices Appendix Table 1. Summary statistics of project playgroups Panel project Non-panel Diff between playgroups project panel and non- playgroups panel (N=245) (N=187) Playgroup characteristics in 2010 Mean Mean Mean (S.D.) Diff (S.E.) Finance Playgroup did not charge student fees (1=Yes) 0.52 (0.49) 0.52 (0.49) 0.00 (0.05) Average wealth of households (z-score) -0.07 (0.99) 0.04 (0.94) 0.10 (0.09) Proportion of block grant spent on: Administrative support 0.14 (0.13) 0.15 (0.13) 0.00 (0.01) Children 0.36 (0.12) 0.36 (0.10) -0.00 (0.01) Infrastructure 0.31 (0.15) 0.31 (0.12) -0.01 (0.01) Teachers 0.15 (0.07) 0.16 (0.07) 0.01 (0.01) Outreach 0.03 (0.05) 0.03 (0.03) -0.00 (0.00) Quality Teachers with post-secondary degree (1=Yes) 0.20 (0.34) 0.20 (0.36) 0.01 (0.03) Teachers with ECED teaching exp. (1=Yes) 0.31 (0.42) 0.33 (0.43) 0.02 (0.04) Market condition No. of kindergartens per 100 children 0.71 (1.45) 0.62 (0.76) -0.09 (0.11) No. of non-project playgroup per 100 children 0.35 (1.45) 0.19 (0.72) -0.16 (0.11) Supplementary services Provides weekly supp. food program (1=Yes) 0.44 (0.48) 0.46 (0.49) 0.02 (0.05) Provides weekly vitamin provision (1=Yes) 0.08 (0.25) 0.12 (0.30) 0.03 (0.03) Ever provided anti-worm medicine (1=Yes) 0.46 (0.49) 0.49 (0.49) 0.03 (0.05) Community support Parental involvement in center (z-score) 0.00 (1.02) -0.00 (0.97) -0.00 (0.10) *** p<0.01, ** p<0.05, * p<0.1 Note: Data from the 2010 Indonesia ECED Survey. See main text for definition of variables. Robust standard errors clustered at the playgroup level. Column of difference is a regression of the variables on an indicator for whether playgroup was a panel playgroup (measured in 2010, 2013 and 2016) or not (only measured in 2010 and open/closed status checked in 2016).   28 Appendix Figure 1. Proportion of all project playgroups that are open between 2010-2016 Note: Plot of proportion of project playgroups that remained open in each year. Former administrators and teachers from project playgroups provided the date of closure. Dashed vertical line in 2013 indicates when the Indonesia ECED Project ended. 29 Appendix Table 2. Project playgroup fees by year and open/closed status in 2016 2010 2013 2016 Center status in Open Closed Open Closed Open Closed 2016: (N = 372) (N=60) (N = 372) (N=60) (N = 372) (N=60) Percent of playgroups NA charging fees 50.57 32.29 70.22 59.22 76.89 Monthly fee in IDR (if charging fees) Min 500 1,000 1,000 5,000 1,500 NA Median 5,000 5,000 10,000 8,000 10,000 Max 35,000 12,000 48,000 25,000 55,000 30 Appendix Figure 2. Distribution of funding for project playgroups (0-300 million Rupiah) from 2010 to 2013 Notes: The distributions in the first to fourth columns are for the closed centers, open centers with no original teacher, one original teacher, and two original teachers, respectively. Those in the first to fourth rows are for the total amount of funding, the amount of funding from governemnt, non-government sources, and community/ parents, respectively. 31 Appendix Figure 2 shows the distributions of funding for project playgroups from 2009 to 2013. Compared with the closed centers, the open centers received more funding in total and some open centers with one or two original teachers received more than 200 million rupiah. However, most playgroups, both those that closed and those that remain open, received less than 50 million Rupiah. Among the open project playgroups, the largest source of funding was from government and community/parents. Appendix Figure 3 presents the distributions of funding for project playgroups from 2013 to 2016. The project playgroups that remained open but without an original teacher received more funding in total. Although most project playgroups that remained open received less than 50 million rupiah, some of those without an original teacher received more than 200 million rupiah. Similar to Figure 2, the largest source of funding was from government and community/parents. Appendix Figure 4 shows the distributions of annual salaries for teacher employed in project playgroups that remained open in 2016. The project playgroups that remained open with two original teachers paid these teachers more, on average, than other project playgroups that remained open. Most of the project playgroups that remained open paid teachers less than 1 million rupiah per year, on average, although some of these with one or two original teachers paid these teachers more than 2 million rupiah per year. Appendix Tables 3 to 5 present the mean and standard deviations of funding and teacher salary payments for project playgroups. These results are aligned with what we find in Appendix Figures 2 to 4. 32 Appendix Figure 3. Distribution of funding for project playgroups (0-300 million rupiah) from 2013 to 2016 Notes: The distributions in the first to third columns are for the open project playgroups with no original teacher, one original teacher, and two original teachers, respectively. Those in the first to fourth rows are for the total amount of funding, the amount of funding from governemnt, non-government sources, and community/ parents, respectively. 33 Appendix Figure 4. Distribution of annual teacher salary (0-6 million rupiah) in 2016 Notes: The distributions in the first to third columns are for the open playgroups with no original teacher, one original teacher, and two original teachers, respectively. 34 Appendix Figure 5. Block grants spent on salary per teacher Note: Histogram shows the amount of block grants spent on teacher salary (million IDR) adjusted for the number of teachers per project playgroup. 35 Appendix Table 3. Descriptive statistics of funding for project playgroups (0-300 million rupiah) from 2010 to 2013 Open playgroups Open playgroups Closed Open playgroups with with no original with one original playgroups two original teachers teacher teacher Mean SD N Mean SD N Mean SD N Mean SD N Total funding in million rupiah 5.42 11.82 20 20.87 35.42 41 31.89 59.75 99 30.906 56.403 85.000 Funding from government in million rupiah 3.71 11.91 20 16.64 35.76 41 26.08 54.39 99 24.856 53.970 85.000 Funding from non-government sources in million rupiah 0.15 0.45 20 1.19 4.85 41 0.39 2.95 99 0.656 3.564 85.000 Funding from community and parents in million rupiah 1.56 3.21 20 3.04 5.40 41 5.42 26.38 99 5.394 16.166 85.000 36 Appendix Table 4. Descriptive statistics of funding for project playgroups (0-300 million rupiah) from 2013 to 2016 Open playgroups Open playgroups Open playgroups with no original with one original with two original teacher teacher teachers Mean SD N Mean SD N Mean SD N Total funding in million rupiah 30.67 71.23 41 12.22 26.90 99 17.00 39.29 85 Funding from government in million rupiah 28.11 72.04 41 8.82 25.53 99 8.88 31.72 85 Funding from non-government sources in million rupiah 0.03 0.16 41 0.37 2.18 99 0.14 0.89 85 Funding from community and parents in million rupiah 2.53 4.83 41 3.03 6.02 99 7.98 25.02 85 37 Appendix Table 5. Descriptive statistics of annual teacher salary (0-6 million rupiah) in 2016 Mean SD N Open playgroups with no original teacher 0.33 0.29 41 Open playgroups with one original teacher 0.43 0.59 99 Open playgroups with two original teachers 0.51 0.50 85 38 Appendix Figure 6. Distance calculations from project playgroups by open/closed status (a) Distance to village center 40 30 frequency 20 10 0 0 2 4 6 Distance to village center (km) Closed in 2016 Open in 2016 (b) Distance to nearest neighboring playgroup 15 10 frequency 5 0 0 2 4 6 8 Distance to nearest neighboring playgroup (km) Closed in 2016 Open in 2016 Note: Histogram shows frequency of closed and open project playgroups by their distance to the village center (a) and to the nearest neighboring playgroup (b). The dashed vertical line indicates the median distance for closed and open project playgroups. 39 Appendix Table 6. Household wealth of children enrolled in project playgroups by open/closed status Wealth z-score of children enrolled in playgroup in 2013 Playgroup open in Playgroup closed in Diff. 2016 2016 Mean -0.016 -0.018 -0.002 (S.D.) or [S.E.] (0.367) (0.335) [0.024] Observations 3,075 212 40 Appendix Table 7. Household wealth of children enrolled in project playgroups by village Wealth z-score of children in villages in 2013 Village has all Village has closed Diff. playgroups open playgroup(s) Mean -0.020 -0.004 0.016 (S.D.) or [S.E.] (0.378) (0.308) [0.014] Observations 2,596 691 No. of villages 168 50 Interpretation of Appendix Table 6 is described in detail in the main text. Appendix Table 7 shows the average household wealth of children enrolled in project playgroups in 2013 by village type. Recall that most villages established two playgroups under the project. The first type of villages had all their project playgroups sustained in 2016 while the second type of villages had at least one closed playgroup in 2016. The mean household wealth z-scores were similar across the two types of villages, with a wealth score of -0.020 S.D. for villages where all project playgroups remained open and a wealth score of -0.004 S.D. for villages with at least one closed project playgroup. 41 Supplementary materials (available upon request) Supplementary Table 1. Alternative logistic regression of project playgroup sustainability on finance characteristics (1) (2) Playgroup is open in 2016 Playgroup is open in with two trained teachers (1 = 2016 (1=Yes) Yes) Finance Amount of total funding in 2009-2013 0.21 -0.01 (units: 100 million IDR) (0.22) (0.06) Received funding from government in 2009- 0.05 0.10 2013 (1=Yes) (0.04) (0.07) Received funding from non-government in -0.11** 0.06 2009-2013 (1=Yes) (0.05) (0.10) Received funding from parents in 2009-2013 0.03 0.08 (1=Yes) (0.05) (0.06) Proportion of annual expenditure on teacher 0.48 -0.40 salaries in 2010 (0.30) (0.52) Change in proportion of annual expenditure 0.43** 0.57 on teacher salaries from 2010-2013 (0.20) (0.38) Log-likelihood -59.25 -152.10 Observations 245 245 * p<0.10, ** p<0.05, *** p<0.01 Note: Robust standard errors clustered by playgroup. Coefficients are marginal effects. Table 3 Alternative explores the relationship between project playgroup sustainability and finance characteristics using a different set of predictors. Specifically, we do not find a significant relationship between project playgroup sustainability and total amount of funding that the playgroup had between 2010-2013, while controlling for whether the project playgroup received any funding from government sources, non-government sources, and parents, as well as proportion of annual expenditure on teacher salaries and change in proportion of annual expenditure on teacher salaries. 42