91321 How can Bulgaria improve its education system? An analysis of PISA 2012 and past results Education Global Practice Europe and Central Asia Region Acknowledgments This report was prepared by Lucas Gortazar, Katia Herrera-Sosa, and Daniel Kutner, with analytical support from Martin Moreno and editing by Amy Gautam. The report is part of the World Bank’s education sector knowledge and advisory services provided to the Government of Bulgaria between 2013 and 2015, led by Plamen Danchev. This report is the first volume of the PISA Country Series conducted by the Education Unit in the World Bank’s Europe and Central Asia Region. The team is deeply grateful to Alberto Rodriguez, Andrea Guedes, Naveed Hassan Naqvi, Cristian Aedo, Omar Arias, and Shagun Mehrotra for their helpful guidance, comments, and assistance. Juan Manuel Moreno and Igor Kheyfets kindly peer reviewed the report; Christian Bodewig, Ismail Radwan, Valerie Morrica, and Allison Berg kindly provided comments. Finally, the team would like to acknowledge the contribution of art director Nicholas Dehaney. Contents Abbreviations and Acronyms Executive Summary 01 1 Why is PISA Important? An Overview of Bulgaria’s Performance on the PISA 05 2 What Determines the Quality of Education in Bulgaria and How Can It Be Improved? 13 The importance of pupils’ origin: the lifelong impact of unequal opportunities 14 Gender disparities 16 Urban-rural disparities 16 Linguistic minorities 17 What can policy makers do to improve education in Bulgaria? 18 A stratified school system: the importance of peers and tracking mechanisms 18 Has the 2007 school autonomy reform worked? 22 Early childhood policies, teacher practices, and the school environment 25 3 Drilling Down Further Into Math and Reading Skills 27 Math skills in PISA 2012 28 Reading skills in PISA 2009 29 Findings and Recommendations 33 4. References 37 Annex 39 Boxes Box 1 Bulgaria’s Education System 08 Box 2 PISA’s Index of Economic, Social, and Cultural Status 11 Box 3 Bulgaria’s 2007 Autonomy Reform 23 Figures Figure 1 PISA scores and public expenditures per pupil 07 Figure 2 PISA 2012 scores for Bulgaria and comparator countries versus ECA and OECD averages 07 Figure 3  Distribution of students by proficiency level in math: (a) Bulgaria’s progress in 2006-2012; (b) Bulgaria and comparators in 2012 09 Figure 4 Student performance in math on PISA 2006-2012 by socioeconomic group 11 Figure 5 Index of equality of opportunities: Bulgaria and other ECA countries, 2012 15 Figure 6 Math and reading performance of Bulgarian students by gender 16 Figure 7 Index of School Social Stratification in PISA 2012-participating countries 18 Figure 8 Evolution of PISA math scores by school type 19 Figure 9  Decomposition of general profiled-vocational school PISA 2012 math score gaps into different factors by student achievement group 21 Figure 10 Decomposition of changes in PISA math scores gaps between 2006 and 2012 into factors and by student achievement group 22 Figure 11 Math contents and process categories in PISA 2012 29 Figure 12 PISA 2012 performance on different math subscales compared to the average math performance 30 Figure 13 PISA 2009 performance on different reading subscales compared to the combined reading performance 31 Tables Table 1 PISA performance byscale for Bulgaria in 2000-2012 10 Table 2 Determinants of achievement in PISA by categories 15 Table 3 Characteristics of students in Bulgaria by language group in 2012 17 Table 4 Rate of affirmative responses on the responsibility over resources and pedagogy of each body, as reported by school principals 23 Abbreviations & Acronyms ESCS ECA Economic, Social, and Cultural Status Europe and Central Asia EU European Union GDP Gross domestic product OECD Organization for Economic Co-operation and Development OLS Ordinary least squares PIRLS Progress in International Reading Literacy Study PISA Programme for International Student Assessment RIF Re-centered influence functions TIMSS Trends in International Mathematics and Science Study UN United Nations UNESCO United Nations Educational, Scientific and Cultural Organization VET Vocational education and training analysis of PISA 2012 and past results 1 Executive Summary Bulgaria’s performance on all three disciplines of the PISA1 2012 was slightly better than its PISA 2000 performance, after having dropped between 2000 and 2006. The improvements post-2006 were greater in reading and math than in science. In the latest PISA (2012), Bulgarian students scored an average of 34 points more on reading and 26 points more on math than in 2006. This represents gains equivalent to almost one year of schooling in reading and a little more than half a year of schooling in math. Despite the recent improvement in achievement, Bulgaria has not made significant progress since 2000 and its performance gap with the OECD accounts for more than one year of schooling. Moreover, around 39 percent of 15-year-old students in Bulgaria are considered functionally illiterate, as they are not able to understand and analyze what they read. Similarly, about 44 percent of Bulgarian students are considered functionally innumerate. The improvements in performance between 2006 and 2012 promoted shared prosperity, but equality of opportunities is still a major challenge. The gains in Bulgaria’s education system between 2006 and 2012 were such that students in the bottom 40 percent of the socioeconomic status made improvements comparable to those of average students (that is, the average score of all students who took the PISA). However, a persistent challenge is the PISA score differentials between students in the highest and lowest socioeconomic quintiles. For instance, in math the difference is approximately 115 points, much higher than OECD standards. Moreover, in Bulgaria, students’ predetermined individual characteristics play a disproportionately high role in explaining PISA scores. Gender, age, and socioeconomic status2 account for almost one-third of students’ differences in reading performance. This reflects the low equality of students’ educational opportunities, as an important share of performance is predefined by students’ backgrounds, potentially limiting social mobility. In fact, disaggregating students’ PISA scores across a number of variables — e.g., location and ethnicity — shows that large inequalities exist in Bulgaria’s education system. Students living in urban areas score as high as 90 points more (or more than two years of schooling) than students in rural areas. There are discrepancies for linguistic minorities as well: Bulgarian-speaking students perform the equivalent of three years of schooling higher in reading and two years of schooling higher in math and science than students who speak a foreign language at home. Peer characteristics and school segregation are the key drivers of the Bulgarian education system’s performance. In Bulgaria, peer characteristics explain more of the differences in PISA test scores than do individual characteristics. That is, the performance of a child on the PISA test depends more on the type of his or her classmates than on his or her own individual factors. This is because the system sorts students into schools populated by other students with similar socioeconomic status, rendering Bulgaria with one of the most stratified educational systems among PISA participating countries. 2 How Can Bulgaria Improve Its Education System? Disparities in performance by school type are large and are exacerbated by the early streaming of students. In Bulgaria, students are streamed into either general profiled or vocational tracks after they take a high-stakes national exam at age 13. The consequence is that most students in general profiled schools, which have a very low share of disadvantaged students, tend to fare quite well. General profiled school students tend to come from families with higher socioeconomic status and interact with similarly better off peers. But over half of Bulgaria’s 15-year-old student population struggles in the worse performing vocational or general non-profiled schools. The analysis of the learning gap between general profiled and vocational education students in Bulgaria shows that socioeconomic status and peer effects explain most of the differences in student outcomes for low-, medium-, and high-achieving pupils. The effects of Bulgaria’s 2007 school autonomy reform on student achievement are mixed and worse than expected. The 2007 governance reform in Bulgaria was a major effort that delegated several responsibilities to school principals—particularly in setting teacher salaries, handling student assessment and admission, undertaking more financial responsibilities, and determining textbook use and course contents. Results of the reform vary by type of autonomy. On one hand, the results show that principals’ greater autonomy in the allocation of resources (such as policies regarding teachers or budget decisions) had a moderately positive impact on all students’ performance (6 PISA points on average), and especially that of low-achieving students. This impact was stronger in urban than in rural areas. On the other hand, the impacts of principals’ greater curriculum and assessment responsibilities on students’ PISA performance were slightly negative, especially in rural areas. Finally, the analysis showed the importance of the quality of educational resources as a key driver of the student performance increase since 2006. An in-depth analysis into math and reading skills shows imbalances in performance in Bulgaria. PISA rotates the in-depth assessment of skills by subject area each time it is administered. PISA 2009 focused on reading, while PISA 2012 focused on math; PISA 2015 will focus on science. Compared with the combined math performance, results in Bulgaria show slightly higher variation across subscale assessments than is found in OECD countries. Students performed better in problems related to space and shape and algebra, and not as well in problems related to data and statistics. In reading, students performed better with more traditional text than with text contained in sample lists, graphs, or diagrams. Moreover, the PISA subscale assessments reveal that Bulgarian students are not good at relating information presented in a text to their own experiences. analysis of PISA 2012 and past results 3 The main areas in which Bulgaria can further improve its educational system involve: Delaying the tracking3 of students to reduce segregation in schools. Bulgaria streams its students 1  into general profiled, general non-profiled, and vocational education schools when they are 13 years old through a high-stakes exam. Existing admission policies on a number of primary schools suggest that this mechanism leads to sorting as early as grade 1. Most countries do this at a later stage, usually when students are 16 years old. A recent World Bank report (2013a) found that the prospective of high-stakes exams creates incentives for parents to invest in private tutoring to help their children increase their scores, leading to sorting among families, which raises important equity concerns. Indeed, PISA score differences between the three streams are fully explained by socioeconomic background and peer effects. Moreover, early tracking hampers the skill development and future long-term employability of students in vocational schools, as they will lack the basic reading and math skills needed for success in a dynamic and rapidly changing labor market. Finally, alternatives to the high-stakes exam that implicitly select students into schools more randomly could further reduce segregation based on students’ abilities. 2  Continue improving the quality of educational resources to ensure that all students learn in an environment with books, lab equipment, and technological hardware and software. The analysis of the improvement in performance in math and reading between 2006 and 2012 shows that the two key drivers were the evolution of students’ socioeconomic status and the improved quality of educational resources. The impact of educational resources was especially important for low-achieving students, indicating a low-hanging fruit for improving the quality and equity of the education system. Continuation of this would include better provision of lab equipment, computer and software materials, library materials, and instructional materials and/or the renovation of buildings and grounds. Encouraging longer pre-primary education for all children. Pre-primary education increases 3  school readiness and has a positive and significant effect on the student achievement of Bulgarian 15-year- olds. This study found that attending at least two years of preschool education raises low achievers’ scores by up to 10 points and the scores of those who speak a different language at home by up to 19 points. In Bulgaria, the pre-primary gross enrollment rate for children aged three to six is 84 percent, but disadvantaged students and minorities still face challenges in accessing this education stage. Promoting early childhood education for all is critical, as cognitive and character skills gaps start opening during early life and inequalities in access to pre-kindergarten perpetuate learning gaps across income groups. 4 How Can Bulgaria Improve Its Education System? 4  Learning from successful schools to improve accountability mechanisms for schools country- wide, particularly in rural areas. There is a need to further understand: (i) why the autonomy reform did not function as expected; (ii) why the reform was more successful in urban areas; and (iii) why PISA scores were positively affected by greater autonomy in the management of school resources, but not by greater autonomy in curriculum development and assessment. Learning from successful schools could help the Government of Bulgaria augment the impact of the reform in rural areas over the coming years. 5  Reevaluating the curriculum and assessment framework to better align student learning to the envisaged country goals. PISA results shed light on Bulgaria’s large discrepancies with other countries within different reading and math skills. PISA results present a good opportunity to engage in an in-depth debate about a curriculum and assessment framework reform, as well as how to better align the education system with national social and economic development goals. Promoting effective classroom management and strengthening teaching practices. The 6  analysis shows that a class that is orderly and has fewer disruptions to students is more conducive to learning and therefore improves PISA scores. The government could use classroom observation methods and international best practices on classroom management to help teachers identify opportunities to improve their performance in the classroom. Teacher development programs could be implemented to improve management techniques in the classroom for the current and future teaching workforce, yielding rapid improvements in the quality of learning. analysis of PISA 2012 and past results 5 Why 1. is PISA important An Overview ? of Bulgaria’s Performance on the PISA 6 How Can Bulgaria Improve Its Education System? Education and skills are critical for the development of both countries and individuals. International evidence suggests that quality of education is one of the most important determinants of long-term economic growth.4 Hanushek and Woessman (2007 and 2012) looked at a wide range of student assessment surveys from 1960 onward, including the Trends in International Mathematics and Science Study (TIMSS), the Programme for International Student Assessment (PISA), and the Progress in International Reading Literacy Study (PIRLS). They estimated that an improvement of 50 points in PISA scores would imply an increase of 1 percentage point in the annual growth rate of GDP per capita.5 Top-quality education systems are also associated with democratic governments. Beyond economic growth, education improves the living standards of individuals, as the more educated are able to acquire more and higher-order skills, making them more productive and employable and extending their labor market participation over their lifetime, which in turn leads to higher earnings and better quality of life. Formal schooling also contributes to development of socio-emotional skills like attention, motivation, self-confidence, and physical and emotional health, all important determinants of socioeconomic mobility. Individuals equipped with more education and skills are better prepared to become civically engaged, improve the democratic capital of their country, and create and make use of opportunities. Education is a key ingredient for reducing inequality and increasing shared prosperity. The analysis of detailed data is critical for understanding the determinants of education quality and can play an important role in shaping effective evidence-based education policy. The PISA database is a great resource in the pursuit of this analysis. PISA is a tool for measuring education quality across countries. Introduced in 2000 by the Organisation for Economic Co-operation and Development (OECD), PISA is a worldwide study of 15-year- old school students’ performance on three different disciplines: math, science, and reading. PISA focuses on the competence of students and their ability to tackle real-life problems in those three disciplines and emphasizes skills that are critical for individuals’ personal and professional development. PISA only assesses students who are in the education system, making it the most internationally comparable snapshot available of a country’s education system. However, if dropout rates are high, the results may not be representative of a country’s cohort of 15-year-olds. PISA’s scoring system is standardized so that the mean score for each discipline among OECD countries in year 2000 is 500 points, with a standard deviation of 100 points. According to OECD, 40 points in PISA is equivalent to what students learn in one year of schooling.6 Bulgaria’s education system (Box 1) was assessed in the PISA rounds of 2000, 2006, 2009, and 2012. Bulgaria’s participation in PISA allows us to benchmark it with other countries, measure the extent to which the country has succeeded in promoting education quality, and gauge whether system inequities have been reduced over time. analysis of PISA 2012 and past results 7 Figure 1 PISA scores and public expenditures per pupil 580 East Asia Korea Hong Kong 540 Japan Europe & Switzerland Central Asia Estonia Finland Netherlands Poland PISA score in mathematics, 2012 Belgium Czech Republic New Zealand Australia Austria 500 Latvia Slovenia France Ireland United Kingdom Denmark Portugal Iceland Norway Lithuania Italy Sweden Croatia Slovak Republic Spain United States Hungary 460 Isreal Serbia Cyprus Western Europe Romania & US/Canada Kazakhstan Bulgaria 420 Thailand Chile Malaysia Mexico Argentina Latin American 380 Indonesia Colombia and Caribbean Peru 340 0 2000 4000 6000 8000 10000 12000 14000 16000 Public expenditures per pupil (in PPP dollars), UNESCO 2012 or latest Source: PISA 2012 and UNESCO 2012. Note: The curve represents a logarithmic approximation of the scatter plots. Figure 2 PISA 2012 scores for Bulgaria and comparator countries versus ECA and OECD averages 530 510 PISA 2012 Scores 490 One year of schooling 470 450 430 410 390 Bulgaria Romania Turkey Serbia ECA EU12 OECD Romania Turkey ECA Serbia ECA EU12 OECD Poland Serbia Bulgaria Turkey EU12 OECD Poland Bulgaria Romania Poland Source: PISA 2012. Math Reading Science 8 How Can Bulgaria Improve Its Education System? Box 1 Bulgaria’s Education System Bulgaria has a population of 7,36 million people (2011), with three large ethnic groups. Those of Bulgarian ethnicity comprise 85 percent of the population; those of Turkish ethnicity, 9 percent; and those of Roma ethnicity, 5 percent. The education system serves over 1.2 million students from pre-primary school through tertiary education. According to UN estimates, Bulgaria’s school-age population is projected to shrink by 10 percent between 2015 and 2030, reflecting the impact of low fertility and migration. Bulgaria’s education system consists of four levels. Pre-primary education is offered to children between three and six (or seven) years old and since 2010, two years of pre-schooling are compulsory, starting from age five. Basic education comprises grades 1 to 8, usually starts at age seven, and is offered by state, municipal, and private school providers. Although lower secondary does not end until the end of grade 8, most students change schools after grade 7, once they take a high-stakes exam that streams students into general profiled schools, vocational education and training (VET) schools, or general non-profiled schools. Upper secondary education is provided by non-profiled, profile-oriented, and technical (vocational) schools. General profiled schools (often referred as “elite schools”) offer general education with additional focus on a selected subject (e.g., a foreign language, mathematics, information and communication technologies (ICT), etc.). General non-profiled schools provide education without extra focus on a given subject, while vocational schools incorporate vocational subjects into the curriculum, often at the expense of time allocated to general curriculum subjects. Education is compulsory for students up to the age of 16. Source: National Statistical Institute and Ministry of Education, Youth and Science, and World Bank (2014). Bulgaria’s performance is slightly below what PISA 2012 recovered to levels slightly above those should be expected given its current level of of 2000, after having dropped between 2000 and public expenditure per student (Figure 1). In 2006 (Table 1). On PISA 2000, Bulgarian students’ addition, Bulgaria’s performance is worse than performance in science was substantially better than expected given its income level. Comparator in reading and math. The drop in 2006 was more countries like Serbia, Romania, and Turkey acute for math and reading, and the recovery in these performed better than Bulgaria on PISA 2012. disciplines was stronger between 2006 and 2012. While a certain level of financial resources is important to ensure access to a minimum standard Bulgaria’s performance is worse than that of of quality, higher levels of expenditures and regional comparator countries (Figure 2). development do not necessarily imply better learning Despite its improved performance since 2006, outcomes. In the case of upper-middle-income Bulgaria’s scores are still lower than those in many countries like Bulgaria, more investment can still Europe and Central Asia (ECA) region countries, and help improve quality, but additional policy efforts its math and reading scores lag 30 points behind the are needed to take education quality to the next level ECA average. While PISA score changes in Bulgaria and make the improvement sustainable. between 2000 and 2012 were not statistically significant, countries such as Turkey and Poland Bulgaria has not made significant progress in carried out sustained and systemic reforms and achievement since 2000. Bulgaria’s performance on saw their scores go up by 30 (Turkey) to 40 (Poland) analysis of PISA 2012 and past results 9 Figure 3 Distribution of students by proficiency level in math: (a) Bulgaria’s progress in 2006-2012; (b) Bulgaria and comparators in 2012 60% 50% 40% n 2006 n 2009 n 2012 30% 20% 10% 0% Below level 2 Level 2 Level 3 Level 4 Level 5+ 100% 90% n Level 5+ 80% n Level 4 70% n Level 3 n Level 2 60% n Below Level 2 50% 40% 30% 20% 10% Bulgaria Turkey Romania Serbia ECA OECD EU12 Poland 0% Source: PISA 2006, 2009, and 2012. 10 How Can Bulgaria Improve Its Education System? 2+2+6=10 4+4+5=13 Table 1 PISA performance by scale for Bulgaria in 2000-2012 2000 2006 2009 2012 Reading 430 402 429 436 Math 430 413 428 439 Science 448 434 439 446 Source: PISA 2000, 2006, 2009, and 20127 points. Finally, Bulgaria’s scores are about 40 points 15-year-old students in Bulgaria score below level 2 below those of EU12 new-member states, and need in math (Figure 3a), meaning that they are not able to to increase by about 60 points to reach the OECD understand and solve simple math problems, severely average in all disciplines (equivalent to one and limiting their development and skill acquisition a half years of schooling). process. The picture is similar for reading: about 39 percent of Bulgarian students are considered Bulgaria has reduced the share of students functionally illiterate. That said, an important part of below basic proficiency levels since 2006, the progress made by Bulgaria since 2006 was due to although it remains high. PISA categorizes scores the improvements of students performing below level in six levels of proficiency; students who score below 2. Countries like Poland have a much lower share of level 2 in the reading and math tests are considered students below level 2 (Figure 3b) and their progress functionally illiterate and innumerate, respectively. in the last decade was also mainly driven by the According to the 2012 data, around 44 percent of improvements of low achievers. analysis of PISA 2012 and past results 11 Figure 4 Student performance in math on PISA 2006-2012 by socioeconomic group 450 440 n Average 430 student n 439 420 428 Math PISA Performance 410 n 400 413  Bottom ESCS 390 40% student  397 380 388 370  360 369 350 2006 2009 2012 The World Bank’s mission has recently been articulated into two main goals: boosting the end of extreme poverty and promoting shared prosperity. The definition of the latter focuses on the income of the bottom 40 percent. This number has been arbitrarily chosen given that: (i) in many low-income countries, the bottom income quintile coincides with the percentage of people in extreme poverty so that this group needed to be expanded; and (ii) this indicator expands this notion to also capture the people considered moderately poor in middle-income countries. Source: Data from PISA 2006, 2009, and 2012. Box 2 PISA’s Index of Economic, Social, and Cultural Status Created by OECD, PISA’s Index of Economic, Social, and Cultural Status (ESCS) is a multidimensional measurement that takes into account information reported by students on their family’s wealth and occupational, educational, and cultural background. It is derived from a combination of three other indexes: (i) an index of the highest occupational status of parents, indicating not only labor market status, but also the type of job held by parents; (ii) an index based on the highest level of parental education in years of schooling; and (iii) an index of family home possessions, which itself consists of a combination of the family’s possessions (such as cars, bathrooms, or technological devices) and educational resources (such as desks, computers, textbooks, the number of other books), as well as the type of cultural possessions (such as the type and genre of books or works of art). The ESCS Index is the most important determinant of student achievement and is therefore crucial for analysis of the quality of education. Source: PISA 2012 results (OECD 2014). 12 How Can Bulgaria Improve Its Education System? Without sustained improvements for all, disadvantaged students are unlikely to experience an increase in their future living standards Improvements since 2006 promoted shared development (see Box 2). Results (in Figure 4) prosperity for the bottom 40 percent, but show that since 2006, the bottom 40 percent of the gap between students of privileged students in terms of socioeconomic status have socioeconomic background and the made advancements in math comparable to those disadvantaged remains high.8 Without of average students (and similar trends are seen in sustained improvements for all, disadvantaged reading and science). However, the differences in students are unlikely to increase their future living math and reading scores between students in the standards. While average score growth is important, highest and lowest quintiles of socioeconomic status it is also crucial to foster improvements among the are 115 and 150 points, respectively (representing bottom 40 percent of a country’s student population. between three and four years of education), while From the PISA data, the OECD’s Index of Economic, the OECD average differences between these income Social, and Cultural Status (ESCS) is used herein as quintiles are 100 points in math and 90 points in a measure of student wealth and level of household reading. analysis of PISA 2012 and past results 13 2.What determines the quality of education in Bulgaria improved? and can it be 14 How Can Bulgaria Improve Its Education System? In this section, we analyze the determinants and drivers of education quality in Bulgaria. We use PISA student achievement as a measure of education quality and relate it to the variables in the PISA student and school questionnaires that can determine quality in an education system. We use different analytical techniques for this purpose, and broadly divide variables into individual and school characteristics, with subgroups of variables within school characteristics: peer characteristics, school resources, and system variables like school autonomy (Table 2).9 The importance of pupils’ origin: the lifelong impact of unequal opportunities PISA results suggest that the opportunities for obtaining a good education are highly unequal in Bulgaria, and mostly depend on students’ background characteristics. As seen in the previous section, the difference in math scores between students in the highest and lowest quintiles of socioeconomic status is very large. Analysis indicates that the importance of certain individual characteristics (gender, age, and socioeconomic status) to students’ performance in Bulgaria is among the highest in the region (Figure 5), explaining 33 percent of the difference in reading achievement,10 and reflecting the low equality of educational opportunities. Disaggregating test scores reveals important differences in the effects of a number of variables, such as gender, school location (rural or urban), and language spoken at home. I love * science analysis of PISA 2012 and past results 15 Figure 5 Index of equality of opportunities: Bulgaria and other ECA countries, 2012 0.35 0.30 More equality of opportunities PISA 2012 Scores 0.25 0.20 0.15 0.10 Kazakhstan Estonia Serbia Russia Poland Lithuania Turkey Slovenia Montenegro Croatia Czech Republic Romania Latvia Hungary Slovak Republic Bulgaria Source: Authors’ calculations based on PISA 2012. Note: The index is the percent of the variance in reading scores explained by the main predeter mined charactristics (age, gender, and socioeconomic status) in a linear regression (Ferreira and Gignoux 2011). Table 2 Determinants of achievement in PISA, by categories Individual Characteristics Age Gender Socio-Economic Status (ESCS Index) Ethnicity Grade Participation in Pre-Primary Education School Characteristics Peer Characteristics School average socioeconomic status Index (ESCS Index) School dropout rate Share of minorities School Resources Quality of Educational Resources (Index) Student-Teacher Ratio Location (Urban or Rural) Parental Engagement Type of school (Public or Private) School Autonomy Responsibility over Curriculum and Assessment (Index) Responsibility over Human and Financial Resources (Index) Source: Greenwald, Hedges and Laine 1996; Hanushek 2009 16 How Can Bulgaria Improve Its Education System? Figure 6 Math and reading performance of Bulgarian students by gender 445 470 n n 440 n n n 450 PISA Reading Perfromance PISA Reading Perfromance 435 n 430 n 430 n n 425 n 410 n n 420 n 390 415 n 410 n 370 n 2000 2003 2006 2009 2012 2000 2003 2006 2009 2012 Source: PISA 2012. n Female Note: Results in 2003 were estimated by linear interpolation. n Male Gender disparities Urban-rural disparities Bulgarian girls outperform boys by almost 70 The disparity between the PISA scores of urban PISA points in reading, while performance in and rural students is high for all three disciplines. math does not vary significantly by gender. In Bulgaria, around 25 percent of PISA-takers live Differences in performance between girls and boys in in rural areas, in municipalities with a population both math and reading have not changed significantly smaller than 15,000. The difference between rural since 2000 (Figure 6). In OECD countries, girls and and urban students’ scores is 89 points in reading boys also perform at similar levels in math. And in and 65 points in math. The difference in math other neighboring countries, girls tend to score higher scores between urban and rural locations is very than boys on the reading scale, as in Bulgaria. For high compared to the ECA average of 27 points. example, girls score 45 points more on reading in As this only provides an absolute number without Serbia, 46 points more in Turkey, and 40 points more taking into account several other differences in the in Romania. Relative to these countries, the difference characteristics of these two subpopulations, the in Bulgaria is very high. In particular, Bulgarian girls’ Annex further explores the key factors behind the enrollment in general profiled schools is higher than urban-rural disparity. Results show that individual boys’: 56 percent of girls study in these programs and peer characteristics as well as school resources versus 40 percent of boys, a streaming process that are the main drivers explaining the differences may be exacerbating the gender gap. between urban and rural students. analysis of PISA 2012 and past results 17 Table 3 Characteristics of students in Bulgaria by language group in 2012 Bulgarian Linguistic speaking minority students students Enrolled in general profiled schools (percent) 51.9 16.0 Live in rural areas (percent) 19.8 44.9 Mother working (percent) 82.0 57.2 Father working (percent) 87.8 76.3 Mother’s education (years) 11.8 9.1 Father’s education (years) 11.5 9.2 Source: PISA 2012. Linguistic minority Linguistic minorities students are In Bulgaria, linguistic minority students lag significantly behind Bulgarian-speaking much less likely students. In 2012, almost 11 percent of students reported speaking a language other than Bulgarian at home. PISA data did not identify which language was spoken by these language minority students, but given the population structure, it is likely that they to be enrolled in general profiled were mostly Turkish and Roma ethnic minorities. Students from linguistic minorities lag behind schools, tend to Bulgarian-speaking students the equivalent of three years of schooling in reading (121 points) and two be concentrated years of schooling in math (75 points) and science (82 points). A more detailed picture shows that the language groups in Bulgaria do not share the same socioeconomic and geographical characteristics (see Table 3). In particular, linguistic minority students more in rural areas, and have are much less likely to be enrolled in general profiled schools, tend to be concentrated more in rural areas, parents who are and have parents who are less educated and less likely to participate in the labor market. Overall, the large gap in educational opportunities between language groups can be summarized by large differences in their socioeconomic backgrounds. less educated 18 How Can Bulgaria Improve Its Education System? Figure 7 Index of School Social Stratification in PISA 2012-participating countries 0.65 0.60 0.55 0.50 0.45 0.40 0.35 0.30 Great Britain Finland Norway Sweden Montenegro Canada Estonia Serbia Russia USA Greece Poland Turkey Slovakia Denmark Latvia Croatia Lithuania Slovenia Czech Republic Germany Romania Hungary Bulgaria Source: Authors’ calculations based on PISA 2012. Note: The index goes from 0 to 1. A higher index indicates a higher correlation between students’ and schools’ socioeconomic status. The figure includes a selected number of PISA countries. What can policy makers Moreover, there is a strong relationship between each do to improve education student’s individual characterstics and those of other students in the same school. in Bulgaria? Social stratification in Bulgarian schools is A stratified school system: the the highest among EU countries (Figure 7). importance of peers and tracking We define the Index of School Social Stratification as the correlation between the PISA student’s mechanisms socioeconomic status and the average school’s Peer effects are a fundamental driver of student socioeconomic status.12 In a world without social achievement in Bulgaria. The previous section stratification (thus an index equal to zero), families studied the importance of individual predetermined from different socioeconomic backgrounds would characteristics, which explained 33 percent of randomly settle across the country and students from students’ differences in reading. However, individual different backgrounds would study together, making characteristics averaged at the school level (i.e., peer schools more diverse. However, households tend to characteristics) explain more of the differences in scores (48 percent) than do individual characteristics. co-locate in neighborhoods with other households This critical finding suggests that a student’s similar to them, and students tend to attend school performance depends more on where he or she attends with peers who have a similar socioeconomic status school than on his or her individual characteristics.11 as a result of spatial inequalities. analysis of PISA 2012 and past results 19 Figure 8 Evolution of PISA math scores by school type 480 470 460 450 n General profiled Math PISA Points n Vocational 440 430 420 410 461 416 475 416 400 2009 2012 Source: PISA 2009 and 2012. Disparities in performance by school type are Delaying student tracking reduces school large and have recently increased. Figure 8 stratification and allows for better opportunities shows the average math scores by type of school for low achievers. Several factors may lead to for the two largest streams of 15-year-old students: segregated schools. Some have to do with the the difference in math scores between general geographic assignment of students to schools profiled school students (comprising 48 percent of (e.g., when wealthier people are concentrated in a the sample in 2012) and vocational school students particular neighborhood). Another factor is the use of (41 percent) increased from 45 to 60 points in math exams to select and stream students at early stages. (equivalent to one and a half years of schooling). The Moreover, parents in high and low socioeconomic situation is similar for reading, with an increase in the groups may have different access to information difference from 68 to 86 points in reading (two years or different priorities when they make schooling of schooling). This indicates that slightly less than decisions. Bulgaria streams its students at age 13 half of Bulgarian children have good opportunities into general profiled, general non-profiled, and in general profiled schools, while most of the other vocational education schools based on a high-stakes half struggle in typically lower-quality vocational exam. Examination of the existing admission or general non-profiled schools. General profiled policies of a number of primary schools suggests schools have a very low share of disadvantaged low- that this mechanism leads to sorting as early as achieving students13 relative to vocational schools. grade 1. A recent World Bank report (2013a) found This means that not only are general profiled students that the prospective of high-stakes exams creates better off in terms of their family background, but incentives for parents to invest in private tutoring, they also have the privilege of interacting with leading to sorting of students and raising important similarly better off peers. equity risks. Most countries with better education 20 How Can Bulgaria Improve Its Education System? Most countries with better education systems stream students at later stages of schooling, usually at age 16. analysis of PISA 2012 and past results 21 Figure 9 Decomposition of general profiled-vocational school PISA 2012 math score gaps into different factors by student achievement group 140 120 100 n Individual 80 characteristics n Peer characteristics 60 n School resources n School autonomy 40 n Unexplained n Actual difference 20 0 -20 High Average Low Middle achiever achiever achiever achiever -40 Source: Authors’ calculations based on PISA 2012. Note: Results decomposition was done using an Oaxaca-Blinder method on RIF-regressions for each quantile of the distribution of performance (Firpo, Fortin and Lemieux 2009). Low achievers are students in the 20th percentile.15 systems stream students at later stages of schooling, individual effects due to their high relation (already usually at age 16. Hanushek and Woessman (2006) explained in this section), it is clear that the ability used previous PISA data to show how early tracking based selection of students through national tests systems lead to a systematic increase in inequality after grade 7, which in practice is implicitly sorting of student performance without affecting average students according to their socioeconomic status, performance levels. This suggests that there are no determines differences between general profiled efficiency gains from introducing early streaming and vocational schools.14 This finding has important of students. policy implications. While little can be done about individual characteristics, policy levers can be Individual and peer socioeconomic used to reduce school segregation and promote characteristics are the major determinants more interaction between children of different of the difference in student achievement backgrounds, which may lead to major improvements between general profiled and vocational in student achievement. schools in Bulgaria. Econometric analysis shows that socioeconomic status and peer characteristics explain most, if not all, of the differences in student outcomes, no matter how students performed in each school (Figure 9). In fact, peer effects appear to be more important than individual characteristics. Although it is difficult to disentangle peer from 22 How Can Bulgaria Improve Its Education System? Figure 10 Decomposition of changes in PISA math scores gaps between 2006 and 2012 into factors and by student achievement group 40 35 30 25 n Individual characteristics 20 n Peer characteristics 15 n School resources n School autonomy 10 n Unexplained n Actual difference 5 Average Low Middle High achiever achiever achiever achiever 0 -5 -10 -15 Source: Authors’ calculations based on PISA 2006 and 2012. Note: Results decomposition was done using an Oaxaca-Blinder method on RIF-regressions for each quantile of the distribution of performance (Firpo, Fortin and Lemieux 2009). Low, middle, and high achievers are students in the 20th, 50th, and 80th percentile, respectively.14 Overall, Has the 2007 school autonomy reform worked? the results By linking student outcomes to school derived from information, PISA data offer a great opportunity to assess Bulgaria’s 2007 school autonomy the governance reform for the first time. In 2007, the Government of Bulgaria engaged in an ambitious reform to reform were decentralize education management from the central to the school level (Box 3). Evidence suggests that it takes time for such autonomy reforms to yield not the game- tangible results, such as an increase in student test scores. Borman et al. (2003) showed that school- changer that based management reforms need about five years to bring fundamental changes at the school level and policy makers about eight years to show up in indicators such as test scores. As the PISA 2012 test was taken five years after the beginning of the 2007 reform, it provides a expected. great opportunity to make an initial assessment of the reform’s impact on Bulgarian student outcomes analysis of PISA 2012 and past results 23 Box 3 Bulgaria’s 2007 Autonomy Reform In 2007, the Government of Bulgaria introduced a decentralization reform to promote greater autonomy in schools with respect to financial and personnel management. The education system became highly decentralized in resource allocation matters after the reform. Schools now have the autonomy to manage their own budgets, a role transferred from the central government to municipalities and from municipalities to schools based on per-capita financing principles. Schools may have their own revenues in addition to those received from the government, although the share of schools’ own revenues in their budgets is modest. School principals have the authority to hire and fire teachers and to decide individuals’ workloads, remuneration, and bonuses within broadly defined central regulations. School principals are hired by the Ministry of Education and its regional structures. However, there is still room for improving the reform’s implementation. Relationships of accountability between principals and parents need further development. School Boards are composed of parents and representatives of the local community, but do not have the legal authority to participate in school decisions, budget preparation, or supervision. Further, student assessments and school- specific assessment data are known to education authorities (central and regional) and schools, but are not disclosed to the public. Assessment results are used to track performance and inform decisions for administrative and pedagogical adjustments, but are not part of a long-term national plan for school improvement, as they are outside the accountability framework. Source: World Bank 2011b Table 4 Rate of affirmative responses on the responsibility of each body over resources and pedagogy, as reported by school principals. Principal School Regional Central (%) Governing Authority Authority Board (%) (%) (%) 2006 2012 2006 2012 2006 2012 2006 2012 Responsibility for teacher hiring 100 99 2 4 6 4 2 2 Responsibility for teacher firing 99 95 3 3 7 2 1 2 Responsibility for teachers’ starting salaries 15 79 0 6 8 2 89 41 Responsibility for teachers’ salary increases 19 90 4 10 10 1 87 25 Responsibility for formulating budget 56 65 4 9 48 26 33 53 Responsibility for budget allocations 83 91 18 31 26 5 20 6 Responsibility for student discipline 37 48 93 93 9 5 55 21 Responsibility for student assessment 24 59 27 39 13 17 91 63 Responsibility for student admission 52 77 47 19 33 20 34 9 Responsibility for textbook use 66 83 61 41 6 2 23 21 Responsibility for course content 20 36 15 7 7 4 90 88 Responsibility for courses offered 18 19 47 55 11 8 88 82 Source: PISA 2006 and 2012 School Questionnaire. The percentage indicates the percentage of principals that reported some responsibility of each administrative body over different resources. 24 How Can Bulgaria Improve Its Education System? and of the reform’s strengths and weaknesses. The Exploratory analysis of changes between model employed to do this decomposes the change in 2006 and 2012 shows little changes in results scores between PISA 2006 (baseline) and PISA 2012 associated with the school autonomy reform, to make a preliminary assessment of the impact of the but highlights the importance of school shift in responsibility from the government to school resources as a driver of improvements for principals. low achievers. Using an approach similar to that followed to identify the factors associated with the Moderate to significant changes in school gap between general profiled and vocational schools, autonomy between 2006 and 2012 allow for the increase in math performance between 2006 and assessment of the reform’s impact. As part of the 2012 is mainly explained by improved socioeconomic PISA, school principals are given a questionnaire conditions and the quality of educational resources in which they respond to questions related to the (Figure 10).16 The improvement in socioeconomic organization of the school, the school’s student conditions (through individual households and peer and teacher bodies, the school instruction and effects) explained most of the performance increase curriculum, the school climate, school policies and for high-achieving students. Similarly, improvements practices, and the school financing. The section in school resources – through increased availability on school policies and practices includes the of quality library materials, lab equipment, and following question: “Regarding your school, who has computer materials – played a crucial role for low considerable responsibility for the following tasks?” achievers (explaining almost half the increase). One For each task, the principal can indicate which of hypothesis for this is that improvements in the school four educational institutions have responsibility learning environment are particularly important for (with more than one response possible): Principals, children who lack materials at home. This finding School Governing Board, Regional Authority, or draws important policy lessons for future decisions. Central Authority. Table 4 displays the percentage The overall effects of the school autonomy reform are of responses given by principals for each of the not statistically significant (see Annex) and suggest educational institutions by specific autonomy that had the reform not been implemented, PISA responsibility in 2006 and 2012. Although this performance would have been essentially the same in does not reflect the exact responsibility of each 2012. Overall, the results derived from the governance educational stakeholder, it displays a major shift reform were not the game-changer that policy makers in responsibility towards principals, mainly in expected. the decision of teacher salaries and in student assessment and admission, and also moderate shifts Although the overall results are limited, the in principals’ responsibilities for budgets, textbook effects of different types of autonomy vary by use, and development of course content. The increase urban and rural settings. The overall impact of the in principals’ decision making allows us to identify reform can be disaggregated by type of autonomy.18 if the reform helped explained the changes in PISA This decomposition includes the interaction of results between 2006 and 2012. autonomy indexes with a rural variable indicator to analysis of PISA 2012 and past results 25 Global evidence shows that providing quality preschool education is important for promoting children’s social, emotional, physical, and cognitive development allow the impact of the reform on rural and urban Early childhood policies, teacher schools to be disentangled (see Annex). On one practices, and the school environment hand, the results show that the shift in autonomy for allocation of resources (such as teacher salaries There is room for policy interventions that have and budget allocation) had a positive and very the power to improve the quality of education. significant impact on all students’ scores (6 points The previous section emphasized how individual and on average), and especially on those of low achievers peer characteristics are an important determinant (11 points). This impact was stronger in urban than of student achievement. In this part of the study, a in rural areas; a possible reason may be better and multilevel analysis of determinants first includes more accountable school administration in urban individual characteristics, peer characteristics, areas. On the other hand, the impact of principals’ and school resources variables (such as quality of greater curriculum and assessment responsibilities educational resources and shortage of teachers). In on students’ performance was slightly negative the next step, the two autonomy measures discussed (although not very significant), outweighing the gains in the previous section are also included in the model made from greater autonomy in resource allocation. of determinants of learning (see the Annex for a The fact that the impact of the reform was higher for summary of results). low-achieving students (especially in urban areas) suggests that greater autonomy allowed principals The analysis finds that early childhood and teachers to focus on those students who lagged education (ECE) has a positive and significant behind or who needed more support. effect on student achievement (see Annex). About 77 percent of 15-year-old students taking the PISA in Bulgaria have more than a year of pre- primary education. This is due to the increased 26 How Can Bulgaria Improve Its Education System? efforts of the Government of Bulgaria to expand the coverage of preschool education during the last decade. Results show that having attended at least a 2-year pre-primary education program increases PISA math scores by an average of 7 points relative to having attended one year or none at all. The effect of ECE is greatest for low achievers (10 points on average) and students who speak a different language at home (19 points on average), while its effect on high achievers is not significant.19 Global evidence shows that providing quality preschool education is important for promoting children’s social, emotional, physical, and cognitive development; it also increases school readiness, which helps learning (Heckman and LaFontaine 2010; Heckman 2008; Engle et al. 2011). Cognitive skills gaps start opening during An orderly school (i.e., one where teachers can teach early life and inequalities in access to early childhood effectively, and students listen to their teachers and perpetuate learning gaps. Given that attendance work well) offers fewer disruptions to students and is in early childhood programs is correlated with more conducive to learning. higher educational attainment, policies improving access to and quality of ECE in Bulgaria for the mostTeaching practices are another important disadvantaged students (who still face challenges in determinant of learning. For instance, effective starting education early) have the highest potential teacher management of a classroom (such as to increase student achievement. This would help keeping the class orderly, getting students to listen, improve the cognitive and social skills of the entirestarting lessons on time, or ensuring that there population, translating into higher human capital are no disruptions) has a positive and significant and productivity and likely contributing to an overall effect on the PISA math score (about 5 points).21 reduction in learning inequality. Nonetheless, changing teaching practices to improve service delivery in education is not straightforward. The school and classroom environment affects Therefore, developing relevant policies to tackle this student achievement. Disciplinary climate issue – such as effectively reforming teacher pre- measures the frequency and severity of disruptions and in-service training or attracting more qualified by students in a school and is an important variable teachers to the teaching force – is a challenge for 20 in explaining students’ academic performance (about the medium and long run. Finally, other school- 6 points on average). Disciplinary climate depends related variables, like class size, were not found to be not only on the student body but also on the social significant in determining student achievement as and managerial abilities of teachers and principals. measured by students’ PISA math scores. analysis of PISA 2012 and past results 27 Drilling 3. down further into math & reading skills 28 How Can Bulgaria Improve Its Education System? PISA offers the opportunity to fully explore one subject area every three years, even though all three subjects are assessed every time PISA is administered. PISA seeks to assess not merely whether students can reproduce knowledge, but also to examine how well they can extrapolate from what they have learned and apply it in unfamiliar settings, both in and outside of school. The detailed test of “subscale” skills of a given subject area is an in-depth assessment with a larger set of questions. The detailed assessment was on reading in 2000 and 2009, on math in 2003 and 2012, and on science in 2006. The 2015 round will focus again on science. Math skills in PISA 2012 The PISA math 2012 subscale assessment measured individuals’ abilities to formulate, employ, and interpret mathematics in a variety of contexts and content areas. In PISA, the concept of mathematical literacy includes: (i) mathematical reasoning; (ii) usage of mathematical concepts, procedures, and facts; (iii) tools to describe, explain, and predict phenomena; and (iv) the role that mathematics plays in the world and the need to make well-founded judgments and decisions needed by constructive, engaged, and reflective citizens. Furthermore, mathematic literacy as defined by PISA is not an attribute that an individual has or does not have; rather, it can be acquired to a greater or lesser extent, and it is required in varying degrees in society. PISA seeks to measure not just the extent to which students can reproduce mathematical content knowledge, but also how well they can extrapolate from what they know and apply their knowledge of mathematics in new situations. PISA’s math framework is a sophisticated tool for connecting students’ mastery of mathematical processes and contents. The math subscale assessment evaluates capacity in four content categories (Figure 11): quantity (incorporates the quantification of attributes of objects, relationships, situations, and entities); uncertainty and data (understanding messages embedded in data, and appreciating variability that is inherent in many real processes); change and relationships (temporary and permanent relations among objects and circumstances); and space and shape (phenomena encountered in patterns, object properties, positions, representations, visual information, navigation, and dynamic interactions). Figure 11 also shows a schematic of the stages faced by a student when solving a real life problem through the mathematical modelling cycle. The action begins with identifying the problem in context and finishes when the results of the problem are found in a context and again are reflected in the problem context. This process involves four skills that PISA defines as “processes,” and were assessed in 2012 as: formulate a mathematical situation according to the concepts and relationships identified; employ mathematical facts, procedures, and reasoning to obtain a result (usually involving calculation, manipulation, and computation); interpret the results in terms of the original problem to obtain the “results in context”; and finally, evaluate the outcomes and their reasonableness in the context of the problem.22 analysis of PISA 2012 and past results 29 Figure 11. Math contents and process categories in PISA 2012 ç problem in context ç formulate mathematical problem ç employ quantity uncertainty and data change and relationships evaluate space and shape results in context Source: OECD 2014. ç interpret mathematical results Reading skills in PISA 2009 Students in Bulgaria performed better in The PISA 2009 subscale assessment of readings problems related to space and shape and skills measured students’ ability to actively, quantity, but not as well in problems related purposefully, and functionally apply reading in to data and statistics (Figure 12). Compared a range of situations. PISA defines reading literacy with the average score of all math subscales, as understanding, using, reflecting on, and engaging Bulgaria’s results show slightly higher variation with written texts to achieve one’s goals, to develop across subscale assessments than is found in OECD one’s knowledge and potential, and to participate in countries. Students successfully solved problems society. Understanding refers to the reader’s ability related to space and shape and quantity, usually to construct meaning from text; using refers to the related to geometry, algebra, and physics. However, kind of reading that is directed toward applying students underperformed when they needed to use information in a text to an immediate task; reflecting their ability to solve data problems or to appreciate means that readers relate what they are reading with variability and uncertainty in real life problems. their thoughts and experiences. Although texts are differentiated in different characteristics (medium, environment, type and format), performance on text format is the only one reported in PISA through two 30 How Can Bulgaria Improve Its Education System? Figure 12. PISA 2012 performance on different math subscales compared to the average math performance 6 Change & Space & Quantity Uncertainty Formulating Employing Performance difference between each content/process Interpreting/ relationships shape and Data Evaluating 4 subscale and the average mathematics scale 2 0 -2 -4 -6 Contents Processes -8 n OECD total Source: PISA 2012. n Bulgaria There is a types: continuous texts (sentences organized into paragraphs, which may fit into even larger structures) need to and non-continuous texts (smaller sentences, usually in sample lists, graphs, diagrams, or catalogues), improve although there are also mixed and multiple texts. Aspects are measured as PISA reading subscales with three categories: access and retrieve (skills the reflection associated with finding, selecting, and collecting information); integrate and interpret (which involves and evaluation understanding the relations between different parts of a text, or making meaning from something that skills in is not stated in the text); and reflect and evaluate (which involves drawing on knowledge, ideas, or values external to the text). Finally, situations intend reading to maximize the diversity of content included in the PISA reading survey; for example, personal, public, educational, and occupational situations are represented. analysis of PISA 2012 and past results 31 Figure 13. PISA 2009 performance on different reading subscales compared to the combined reading perfor mance 10 Continuous Non- Access Integrate Reflect Texts Continuous & retrieve & Interpret & Evaluate Performance Difference between each content/process Texts 5 subscale and the combined mathematics scale 0 -5 -10 -15 Texts Aspects n OECD total Source: PISA 2009. n Bulgaria 32 How Can Bulgaria Improve Its Education System? 8-4+1=5 120-10=110 The 2009 subscale assessment for reading revealed that Bulgarian students have a better understanding of continuous text compared with non-continuous text, while there is a need to improve their reflection and evaluation skills in reading. Comparing the reading subscale results with the average score across all reading subscales, Bulgaria shows much more variation across subscales compared with OECD countries, which means there is large room for improvement in some subscales. In particular, students perform better with more traditional texts rather than texts contained in sample lists, graphs, or diagrams. Moreover, students’ ability to relate their own experiences to the text is weak, reflecting a disconnect between what students learn and their ability to apply this knowledge in real life situations. analysis of PISA 2012 and past results 33 4. Findings recommendations & 34 How Can Bulgaria Improve Its Education System? After a drop between 2000 and 2006, Bulgaria’s PISA scores improved in all three disciplines. To sustain the recent success, new policies are required. A large share of the improvement since 2006 is explained by the improvement in students’ socioeconomic status, which translated into better test scores, as well as the better quality of educational resources. It is now necessary to devise a new set of effective policies to continue narrowing the gap in scores with OECD and other countries in the region. Investment in educational resources is important to ensure minimum standards, but is not sufficient to sustain continuous improvement. Although it is difficult to affect students’ predetermined characteristics in the short term, there is still an important role for policy. In Bulgaria, the difference in performance between students in the bottom and top socioeconomic quintiles is much larger than in OECD countries. The significance of predetermined factors in affecting students’ educational performance can be discouraging, as these factors generally take time, often generations, to improve. Inequality of educational opportunities in Bulgaria is the highest in the region and the EU. Disadvantaged groups, such as rural populations and linguistic minorities, perform much worse on the PISA than urban populations and Bulgarian-speaking students. Moreover, the performance gap between girls and boys on the PISA reading score is the highest in the region. An assessment of Bulgaria’s 2007 school autonomy reform shows little impact. This report is the first analysis of the impact of Bulgaria’s school-based management reform, which shifted more responsibility to school principals. The results show that overall the results have been more limited than expected, especially given the amount of effort expended on the 2007 reform. A detailed analysis shows that principals’ greater autonomy over curriculum and assessment policies had a slight negative impact on Bulgaria’s 2012 PISA math scores, while their greater autonomy over management of resources (teachers and budget allocation) had a positive impact. The impact of the reform was higher in urban schools, suggesting better and more accountable school administration in urban areas. Overall, the results indicate the need to further improve the management capacity of principals in rural areas while also strengthening accountability mechanisms. Students performed better on problems related to space and shape and quantity, and not as well on problems related to data and statistics. Compared with the average math performance of all subscales, Bulgaria’s results show slightly higher variation across subscale assessments than is found in OECD countries. Students successfully solved problems related to space and shape and quantity, usually related to geometry, algebra, and physics. However, students underperformed when they needed to use their ability to solve data problems or to appreciate variability and uncertainty in real life problems. analysis of PISA 2012 and past results 35 Bulgarian students have a better understanding secondary schools, like streaming students at the of continuous text than of non-continuous text, end of compulsory education (age 16), could raise and there is a need to improve their reflection the overall education quality of the less favored and evaluation skills in reading. Comparing the without lowering average performance. reading subscale results with the average reading performance of all subscales, Bulgaria shows much 2  Continue to improve the quality of more variation across subscales compared with educational resources to ensure that all OECD countries, which means there is large room students learn in an appropriate environment for improvement in some subscales. In particular, of books, libraries, lab equipment, and students perform better with more traditional texts technological resources. The analysis of than with texts contained in sample lists, graphs, the improvement in performance in math and or diagrams. Moreover, PISA reveals students’ reading between 2006 and 2012 shows that the weaknesses in relating their own experiences to the two key drivers were the evolution of students’ text, reflecting a disconnect between what they learn socioeconomic status and the improved quality of and their ability to apply that knowledge in real life educational resources. The impact of educational situations. resources was especially important for low- achieving students, indicating a low-hanging If adequate policies are pursued, Bulgaria is fruit for improving the quality and equity of the likely to succeed in increasing the equality education system. Continuation of this would of opportunities to achieve its “Learning For include better provision of lab equipment, All” goals. With this in mind, six main policy computer and software materials, library recommendations arise as a result of this study: materials, and instructional materials and/or the renovation of buildings and grounds. 1 Delay the tracking of students into different types of schools as it leads to school 3  Expand preschool education for the most stratification with no benefits. School disadvantaged students, as analysis shows stratification – the concentration of students with it is especially beneficial for the less favored. similar socioeconomic status in the same schools The study found that the expansion of preschool – is a result of the inequalities in the Bulgarian education to at least two years raises low achievers’ education system combined with use of a high- and minorities’ scores by up to 10 and 19 points, stakes exam that channels students into different respectively (even after taking into account other schools according to their socioeconomic status. As relevant individual and school factors). Universal a consequence, disadvantaged students suffer not preschool education would provide a great only from their own situation but are also penalized opportunity to effectively narrow the skills gap by having to interact with similarly disadvantaged from the early stages of children’s lives. peers. Thus, it is plausible that the implementation of adequate selection mechanisms for students in 36 How Can Bulgaria Improve Its Education System? 4  Learn from successful schools to improve 6  Promote effective classroom management accountability mechanisms for schools and strengthen teaching practices. The country-wide, particularly in rural areas. analysis shows that a class that is orderly, with There is a need to further understand: (i) why the fewer disruptions to students, is more conducive autonomy reform did not function as expected; to learning and therefore improves PISA scores. (ii) why the reform was more successful in urban The government could use classroom observation areas; and (iii) why PISA scores were positively methods and international best practices on affected by greater autonomy in the management classroom management to help teachers identify of school resources, but not by greater autonomy opportunities to improve their performance in in curriculum development and assessment. the classroom. Teacher development programs Learning from successful schools could help the could be implemented to improve management Government of Bulgaria augment the impact of the techniques in the classroom for the current reform in rural areas over the coming years. and future teaching workforce, yielding rapid improvements in the quality of learning. 5  Reevaluate the curriculum and assessment framework to better align student learning to the envisaged country goals. The PISA full assessment analysis derives important lessons for policy makers in Bulgaria. Results shed light on the large discrepancies (as compared to other countries) within reading and math skills. PISA results present a good opportunity to engage in an in-depth debate about a curriculum and assessment framework reform, as well as how to better align the education system with national social and economic development goals. analysis of PISA 2012 and past results 37 References Amermueller, A. 2004. “PISA: What Makes the Difference? Explaining the Gap in Pisa Test Scores between Finland and Germany.” ZEW Center for European Economic Research Discussion Paper No. 04-004. Barrera-Osorio, F., V. Garcia-Moreno, H.A. Patrinos, and E. Porta. 2011. “Using the Oaxaca-Blinder Decomposition Technique to Analyze Learning Outcomes Changes Over Time: An Application to Indonesia’s Results in PISA Mathematics.” World Bank Working Paper 5584. World Bank, Washington, DC. Borman, G.D., G.M. Hewes, L.T. Overman, and S. Brown. 2003. “Comprehensive School Reform and Achievement: A Meta-Analysis.” Review of Educational Research 73 (2): 125-230. Engle, P., L. Fernald, H. Alderman, J. Behrman, C. O’Gara, A. Yousafzai, M. Cabral de Mello, M. Hidrobo, N. Ulkuer, I. Ertem, S. Iltus, and Global Child Development Steering Group. 2011. “Strategies for reducing inequalities and improving developmental outcomes for young children in low-income and middle-income countries.” The Lancet 8 October Vol. 378, Issue 9799: 1339-1353 ). DOI: 10.1016/S0140-6736(11)60889-1. Ferreira, H.G., and J. Gignoux. 2011. “The Measurement of Educational Inequality: Achievement and Opportunity.” IZA Discussion Paper No. 6161. Firpo, S., N. Fortin, and T. Lemieux. 2009. “Unconditional Quantile Regressions.” Econometrica Vol. 7, No 3: 953-973. Greenwald, R., L. V. Hedges, and R. Laine. 1996. “The Effect of School Resources on Student Achievement.” Review of Educational Research Vol. 66, No. 3 (Autumn): 361-396. Hanushek, E. 2009. “School policy: Implications of recent research for human capital investments in South Asia and other developing countries.” Education Economics 17(3): 291-313. Hanushek, E. 2010. “The High Cost of Low Educational Performance. The long-run economic impact of improving PISA outcomes.” OECD Publications. Hanushek, E., and L. Woessmann. 2006. “Does Educational Tracking Affect Performance and Inequality? Differences-in-differences evidence across countries.” The Economic Journal Vol. 116, Issue 510: C63-C76. Hanushek, E., and L. Woessmann. 2007. “The Role of Education Quality in Economic Growth.” World Bank Policy Research Working Paper 4122. World Bank, Washington, DC. 38 How Can Bulgaria Improve Its Education System? Hanushek, E., and L. Woessmann L. 2012. “Do Better Schools lead to more growth? Cognitive skills, economic outcomes, and causation.” Journal of Economic Growth Vol. 17: 267-321. Heckman, J. 2008. “Schools, skills, and synapses.” Economic Inquiry 46(3): 289-324. Heckman, J., and P. LaFontaine. 2010. “The American High School Graduation Rate: Trends and Levels.” Review of Economics and Statistics 92(2): 244–262. OECD. 2012. PISA 2009 Technical Report. Paris: OECD. Retrieved April 10, 2014 from http://www.oecd. org/pisa/pisaproducts/50036771.pdf OECD. 2014. PISA 2012 Results: What Students Know and Can Do. Paris: OECD. Retrieved April 10, 2014 from http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-I.pdf Sala-i-Martin, X., G. Doppelhofer, and R.I. Miller. 2004. “Determinants of long-term growth: A Bayesian averaging of classical estimates (BACE) approach.” American Economic Review 94 (4), 813-835. World Bank. 2011a. “Assessing the Quality of Education in Bulgaria using PISA 2009 data.” World Bank, Washington, DC. World Bank. 2011b. “Benchmarking School Autonomy and Accountability in Selected European Countries.” World Bank, Washington, DC. World Bank. 2013a. “Improving the transition between lower and upper secondary schools in Turkey: Recommendations for the design of a more equitable system.” World Bank. Processed Papers (Unpublished). World Bank, Washington, DC. World Bank. 2013b. “Promoting Excellence in Turkey’s Schools.” World Bank, Washington, DC. World Bank. 2014. “Bulgaria: SABER Workforce Development.” World Bank, Washington, DC. analysis of PISA 2012 and past results 39 Annex The analytical approach used in Section 2 of this report is based on the Firpo, Fortin, and Lemieux (2009) methodology. Typically, the literature on decomposition of student scores in PISA through groups (Amermueller 2004) and years (Barrera et al. 2011) has focused on the mean differences, with little attention to what happens at the tails of the distribution. The Firpo, Fortin, and Lemieux (FFL) method allows one to decompose gaps in student performance not only for the mean but also for other statistics of the distribution. Traditionally, the problem with quantile regressions has been that the law of iterated expectations does not apply, thus making it impossible to interpret the unconditional marginal effect of each independent variable on a student’s performance. However, recent econometric techniques, such as the one proposed by FFL, have solved this methodological difficulty. The FFL technique is based on the construction of re-centered influence functions (RIF) of a quantile of interest, , as a dependent variable in a regression: ⌧ − D(I  q⌧ )) RIF (I ; q⌧ ) = q⌧ + f I ( q⌧ ) where is an indicator function and is the density of the marginal distribution of scores. A crucial characteristic of this technique is that it provides a simple way of interpreting the marginal impact of an additional unit of a certain factor on students’ PISA scores. Once the unconditional quantile regression has been computed for different quantiles of the distribution, the results can be decomposed following the Oaxaca-Blinder approach. 40 How Can Bulgaria Improve Its Education System? Table A. 1. Decomposition of urban-rural PISA math score gaps by student achievement groups. Variables Average Percentile 20 Percentile 50 Percentile 80 Rural 400.2*** 307.0*** 401.2*** 449.2*** (6.980) (17.49) (6.230) (9.039) Urban 459.1*** 397.8*** 451.3*** 542.5*** (5.518) (7.648) (4.715) (12.50) Difference -58.87*** -90.72*** -50.10*** -93.33*** (8.897) (19.09) (7.813) (15.43) Unexplained -38.70*** -52.55** -30.34*** -53.32*** (9.693) (21.38) (9.874) (19.15) Explained -20.17* -38.17 -19.76** -40.01*** (10.39) (25.32) (10.07) (15.08) Individual Characteristics -16.05*** -40.66*** -12.14*** -14.03*** (3.084) (9.369) (2.924) (5.130) Peer Characteristics 1.814 19.35 -3.437 -22.04 (9.272) (24.80) (9.199) (13.65) School Resources -4.670 -14.78 -3.335 -2.714 (5.128) (14.11) (5.236) (6.678) Autonomy -1.263 -2.083 -0.851 -1.225 (1.875) (4.290) (1.768) (2.383) Constant 17.62 77.70 -136.9 156.9 (161.0) (489.4) (174.7) (268.2) Observations 4,501 4,501 4,501 4,501 Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variable effects are grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, and socioeconomic status), peer characteristics (socioeconomic status, school dropouts, and minorities at school), school resources (school owner ship, location, quality of educational resources, teacher shortage, and parental pressure), and school autonomy (autonomy in resources, and autonomy in curriculum and assessment). analysis of PISA 2012 and past results 41 Table A.2. Decomposition of general-vocational PISA math score gaps by student achievement group Variables Average Percentile 20 Percentile 50 Percentile 80 Year 2012 444.9*** 373.7*** 439.3*** 522.1*** (4.532) (6.902) (3.789) (9.214) Year 2006 420.0*** 337.2*** 417.4*** 501.3*** (6.251) (3.469) (6.773) (9.796) Difference 24.88*** 36.50*** 21.86*** 20.81 (7.721) (7.724) (7.760) (13.45) Explained 17.20*** 29.84*** 13.15** 26.91** (6.601) (10.42) (5.725) (13.39) Unexplained 7.680 6.658 8.710 -6.102 (5.654) (9.553) (6.281) (10.79) Individual Characteristics -0.464 1.295 0.117 -2.025 (1.125) (2.597) (1.022) (1.995) Peer Characteristics 9.348** 8.612* 7.611** 21.61** (4.647) (4.542) (3.790) (10.71) School Resources 6.980*** 16.68*** 4.277 6.838 (2.709) (6.176) (2.614) (6.170) Autonomy 1.333 3.252 1.141 0.487 (2.467) (5.117) (2.384) (5.442) Constant -89.58 -443.1*** 55.61 -418.0* (114.6) (170.3) (139.0) (228.5) Observations 8,749 8,749 8,749 8,749 Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variables effects are grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, and socioeconomic status), peer characteristics (socioeconomic status, school dropouts, and minorities at school), school resources (school owner ship, location, quality of educational resources, teacher shortage, and parental pressure), and school autonomy (autonomy in resources, and autonomy in curriculum and assessment). 42 How Can Bulgaria Improve Its Education System? Table A.3. Decomposition of 2006-2012 PISA math score gaps by student achievement group Variables Average Percentile 20 Percentile 50 Percentile 80 Year 2012 444.9*** 373.7*** 439.3*** 522.1*** (4.532) (6.902) (3.789) (9.214) Year 2006 420.0*** 337.2*** 417.4*** 501.3*** (6.251) (3.469) (6.773) (9.796) Difference 24.88*** 36.50*** 21.86*** 20.81 (7.721) (7.724) (7.760) (13.45) Explained 17.20*** 29.84*** 13.15** 26.91** (6.601) (10.42) (5.725) (13.39) Unexplained 7.680 6.658 8.710 -6.102 (5.654) (9.553) (6.281) (10.79) Individual Characteristics -0.464 1.295 0.117 -2.025 (1.125) (2.597) (1.022) (1.995) Peer Characteristics 9.348** 8.612* 7.611** 21.61** (4.647) (4.542) (3.790) (10.71) School Resources 6.980*** 16.68*** 4.277 6.838 (2.709) (6.176) (2.614) (6.170) Autonomy 1.333 3.252 1.141 0.487 (2.467) (5.117) (2.384) (5.442) Constant -89.58 -443.1*** 55.61 -418.0* (114.6) (170.3) (139.0) (228.5) Observations 8,749 8,749 8,749 8,749 Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variables effects are grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, and socioeconomic status), peer characteristics (socioeconomic status, school dropouts, and minorities at school), school resources (school owner ship, rural, quality of educational resources, and parental pressure), and school autonomy. analysis of PISA 2012 and past results 43 Table A.4. Decomposition of 2006-2012 PISA math score gaps by student achievement group, detailed autonomy variables Variables Average Percentile 20 Percentile 50 Percentile 80 Year 2012 444.9*** 373.7*** 439.3*** 522.1*** (4.532) (6.902) (3.789) (9.214) Year 2006 420.0*** 337.2*** 417.4*** 501.3*** (6.251) (3.469) (6.773) (9.796) Difference 24.88*** 36.50*** 21.86*** 20.81 (7.721) (7.724) (7.760) (13.45) Unexplained 7.680 6.658 8.710 -6.102 (5.654) (9.553) (6.281) (10.79) Explained 17.20*** 29.84*** 13.15** 26.91** (6.601) (10.42) (5.725) (13.39) Individual Characteristics -0.464 1.295 0.117 -2.025 (1.125) (2.597) (1.022) (1.995) Peer Characteristics 9.348** 8.612* 7.611** 21.61** (4.647) (4.542) (3.790) (10.71) School Resources 6.980*** 16.68*** 4.277 6.838 (2.709) (6.176) (2.614) (6.170) Autonomy Curriculum -1.891 -2.598 -2.242 -2.278 (1.495) (3.105) (1.428) (3.574) Autonomy Curriculum (Interaction with Rural) -0.996 -2.118 -0.555 -2.055 (0.698) (1.743) (0.629) (1.470) Autonomy Resources 7.008*** 13.95*** 7.304*** 10.02* (2.360) (4.572) (2.172) (5.851) 44 How Can Bulgaria Improve Its Education System? Variables Average Percentile 20 Percentile 50 Percentile 80 Autonomy Resources (Interaction with Rural) -2.788** -5.981* -3.367*** -5.203** (1.271) (3.053) (1.244) (2.342) Constant -89.58 -443.1*** 55.61 -418.0* (114.6) (170.3) (139.0) (228.5) Observations 8,749 8,749 8,749 8,749 Note: Robust standard error in parentheses and clustered at the school level. *** p<0.01, **p<0.05,*p<0.1. Variables effects are grouped and include individual characteristics (age, gender, grade, language at home, participation in ECE, socioeconomic status), peer characteristics (socioeconomic status, school dropouts, minorities at school), school resources (school ownership, rural, quality of educational resources, parental pressure), and school autonomy. Table A.5. Share of variation in mathematics scores: multilevel models Model 1 Model 2 Individual characteristics (gender, ESCS, Grade YES YES School characteristics (Disciplinary climate, peer characteristics, and teacher shortage) YES YES System characteristics (autonomy variables— autonomy in resources and in curriculum and assessment) YES Explained variation (%) 0.52 0.53 Source: PISA 2012. analysis of PISA 2012 and past results 45 Table A 6. Determinants of math performance: a multilevel approach Model 1 Model 2 Model 3 ESCS 7.57*** 7.58*** (1.23) (1.23) Kindergarten 6.89*** 6.91*** (2.41) (2.41) Female -15.42*** -15.45*** (2.07) (2.07) Foreign language at home -17.22*** -17.53*** (3.95) (3.95) Age 6.34* 6.38* (3.47) (3.47) Mathematics anxiety -20.04*** -20.03*** (1.04) (1.04) Sense of belonging 2.06* 1.99* (1.15) (1.15) ESCS-school 44.16*** 43.20*** (5.12) (5.06) Teacher shortage 6.18 4.71 (5.71) (5.62) Student-teacher ratio -0.15 -0.12 (0.13) (0.13) Student-teacher relations -4.98*** -4.96*** (1.09) (1.08) 46 How Can Bulgaria Improve Its Education System? Table A.6. Determinants of math performance: a multilevel approach Model 1 Model 2 Model 3 Teacher support -1.87 -1.84 (1.21) (1.21) Disciplinary climate 6.05*** 6.04*** (1.25) (1.25) Grade 19.71*** 19.71*** (4.14) (4.14) Classroom management 5.38*** 5.41*** (0.97) (0.97) Rural -6.72 -6.05 (7.49) (7.33) Educational Resources 3.84*** 3.59*** (1.33) (1.31) - Program 2 -9.12 9.23 (11.31) (11.28) Program 3 7.68 7.52 (10.37) (10.34) Program 4 -2.24 -2.42 (10.74) (10.68) Autonomy curriculum -7.09** (3.10) Autonomy resources 5.80** (2.85) analysis of PISA 2012 and past results 47 Table A.6. Determinants of math performance: a multilevel approach Model 1 Model 2 Model 3 _cons 429.75*** 177.29*** 174.84*** (5.19) (59.50) (59.48) ICC (Intraclass correlation, % of variance attributable to schools) 0.58 0.30 0.28 Source: PISA 2012 Bulgaria. Note: Multilevel models are able to analyze data in nested structure (students within classrooms, within schools) and allow correlation of observations within clusters. For this exercise, we use a random coefficient model at the school level (disciplinary climate). ECE is measured as two years of pre-primary education, and the baseline is one year or less of pre-primary education. Standard errors in parenthesis, *** p<0.01, **p<0.05,*p<0.1 48 How Can Bulgaria Improve Its Education System? Endnotes 1 Socioeconomic status is 7  Ferreira and Gignoux 11  Although results show the measured in PISA with the (2011) propose a measure weight of peer effects to be OECD’s Economic, Social, and of educational opportunity more important than that of Cultural Status Index (ESCS). using the share of variance in individual socioeconomic test scores that is explained characteristics, this should be 2  Tracking of students refers by individual predetermined interpreted with caution, as the to separating students into circumstances. If a significant high correlation between them different academic paths. share of the results is explained indicates that both matter. by these characteristics, then 3 See Sala-i–Martin, the equality of opportunities is 12 Decomposition included Doppelhofer, and Miller (2004). low. individual characteristics, peer characteristics, school 4  See Hanushek and Woessman 8  In fact it depends on with whom resources, and autonomy. (2007) and Hanushek (2010). he or she attends school. Student and peer characteristics Using these tests as measures were the most important of cognitive skills of the 9  See World Bank (2013b). characteristics in the regression population, they show that (full results can be found in countries that had better quality 10 According to PISA data, Table A.2 in the Annex). By of education in the 1960s a student is classified as a decomposing differences, experienced faster economic disadvantaged low achiever if he one often finds that one of the growth during the years 1960- or she is in the bottom quarter explanatory factors is negative 2000, controlling for other of the PISA ESCS Index in a or higher than the actual factors. country and performs in the difference, meaning that other bottom quarter of students from factors outweighed their impact. 5  PISA 2009 Technical Report all countries/countries, after (OECD 2012). accounting for socioeconomic 13 In this analysis, parental and status. Only 2.8 percent of teacher engagement in the 6  Note: Countries that students in general profiled school community were used participated only once in schools are disadvantaged as proxies to control for school PISA between 2000 and 2012 low achievers, while the figure accountability. were not considered for the increases to 12.5 percent in ECA average trend. Linear vocational schools. 14 By decomposing differences, interpolations were made for one often finds that one of the Albania, Bulgaria, and Romania explanatory factors is negative in missing years. or higher than the actual difference, meaning that other factors outweighed their impact. analysis of PISA 2012 and past results 49 15  OECD aggregates all the autonomy measurements shown in Table 4 into two indexes: an index that relates to autonomy in resource allocation (RESPRES), such as teachers and budget preparation, and an index that relates to curriculum and assessment policies (RESPCURR), such as course content, textbooks, or assessment policies. 16  Low achievers were classified as those students at the bottom 20% of the learning distribution. 17  The Disciplinary Climate Index is derived from students’ reports on how often the followings happened in their lessons: (i) students don’t listen to what the teacher says; (ii) there is noise and disorder; (iii) the teacher has to wait a long time for the students to quiet down; (iv) students cannot work well; and (v) students don’t start working for a long time after the lesson begins. Education Global Practice Europe and Central Asia Region