90742 Education quality and opportunities for skills development in Albania An analysis of PISA 2000-2012 results Education Global Practice Europe and Central Asia Region Acknowledgments This report was prepared by Lucas Gortazar and Daniel Kutner (Education Sector, Europe and Central Asia Region, the World Bank), with support from Flora Kelmendi (Education Sector, Europe and Central Asia Region, the World Bank), Kozeta Diamanti (Program Assistant, Albania Country Office, the World Bank), Martin Moreno (Consultant) and editing by Amy Gautam (Consultant). The report is part of the World Bank’s technical assistance provided to the Government of Albania, led by Keiko Inoue (Senior Education Specialist, Europe and Central Asia Region, the World Bank). This report is the third study of the PISA Country Series conducted by the Education Unit in the World Bank’s Europe and Central Asia Region. The team is deeply grateful to Alberto Rodriguez, Andrea Guedes, Cristian Aedo, Omar Arias, and Shagun Mehrotra for their helpful guidance, comments, and assistance. Maria Ramirez and Soren Nellemann kindly peer reviewed the report. Finally, the team would like to acknowledge the contribution of art director Nicholas Dehaney. Contents Abbreviations and Acronyms Executive Summary 01 1 Why is PISA Important? An Overview of Albania’s Performance on PISA 07 2 Poor Quality and Persistent Inequities 15 Inequities in Albania: Which students lag behind? 18 Gender disparities 21 Urban-rural disparities 21 Public and private schools 22 The Albanian diaspora 22 The importance of learning from peers 24 Early childhood policies 25 How Do Classroom-Level Practices Affect Learning? 26 Math skills in PISA 2012 26 Reading skills in PISA 2009 29 Teaching practices and learning strategies in reading 31 Recommendations 33 3 References 36 Annex 38 Boxes Box 1 Albania’s Education System 10 Box 2 PISA’s Index of Economic, Social, and Cultural Status and PISA Data Validity in Albania 17 Box 3 Sample PISA 2012 Math Problem: Which Car? 27 Box 4 Sample PISA 2009 Reading Question: Labor 30 Figures Figure 1 PISA scores and GDP per capita 09 Figure 2 Distribution of students by proficiency level in math: (a) Albania’s progress from 2000-2012; (b) Albania and comparators in 2012; (c) Evolution of math scores in Albania from 2000-2012 12 Figure 3 PISA 2012 scores for Albania and comparator countries versus ECA and OECD averages 13 Figure 4 Distribution of PISA 2012 reading scores in Albania and the OECD 18 Figure 5 Index of equality of opportunities: Albania and other ECA countries, 2009 19 Figure 6 Albanian students’ performance on math and reading in PISA 2000-2009 by socioeconomic group 20 Figure 7 Gender gap in reading, PISA 2009, selected countries 22 Figure 8 The PISA 2009 performance of the Albanian diaspora 23 Figure 9 Index of School Social Stratification in PISA-participating countries, 2009 25 Figure 10 Math contents and process categories in PISA 28 Figure 11 PISA 2012 performance on different math subscales compared to each group’s average performance 29 Figure 12 PISA 2012 performance on different reading subscales compared to each group’s average reading performance 31 Figure 13 Decomposition of the reading performance gap between villages/towns and cities (PISA 2009) 32 Tables Table 1 Albania’s PISA performance by discipline, 2000-2012 11 Table 2 Difference in PISA 2009 performance between top and bottom socioeconomic quintile groups in Albania 20 Table 3 Characteristics of student by school type in 2009 23 Abbreviations & Acronyms ESCS Economic, Social, and Cultural Status ECA GDP Europe and Central Asia Gross domestic product LSMS Living Standards Measurement Survey NES National Education Strategy OECD Organisation for Economic Co-operation and Development OLS Ordinary least squares PIRLS Progress in International Reading Literacy Study PISA Programme for International Student Assessment TIMSS Trends in International Mathematics and Science Study UN United Nations UNESCO United Nations Educational, Scientific and Cultural Organization An analysis of PISA 2000-2012 results 1 Executive Summary Albanian students improved their performance on PISA1 reading, math, and science assessments between 2000 and 2012, a period over which access to upper secondary education expanded dramatically. The improvements are equivalent to a quarter to half a year of schooling in math and science and more than one year of schooling in reading. The improvements were realized despite substantial increases in enrollment and the attendant changes in the PISA sample composition. The gross enrollment rate in upper secondary education was only about 40 percent in 2000, but increased markedly to 72 percent in 2009 and 83 percent in 2012. A spike in enrollment usually results in lower average student performance due to the inclusion of poorer and traditionally excluded students. In the context of a rapidly expanding education system, it is remarkable that Albania managed to improve its average PISA scores. It is even more notable that the improvement came from significant advancements made by low-achieving students. Albania’s improved PISA scores coincide with the launch of intensive reform efforts in its education sector. The Government of Albania initiated the education sector strategy formulation process in 2002, which resulted in the 2004 passage of the National Education Strategy (NES) and the first attempt to develop a long-term roadmap for the sector. The NES served as the catalyst for a range of reforms, including: improved teacher recruitment, compensation, and management; a revised curriculum for basic and general upper secondary education; enhanced transparency and accountability through reform of the Matura, the national student assessment; reduced price and improved textbook quality through a reformed procurement process; and provision of textbook subsidies to the poorest households. While it is beyond the scope of this report to determine a causal relationship between the range of policy reforms and improved PISA scores, this report shows that socioeconomic variables played an important role. Explanatory analyses show that the effect of education variables cannot be disentangled from the effect of socioeconomic variables. Furthermore, with economic growth averaging 4.9 percent per year from 2000-2012, a viable explanation is that the general improvement in economic conditions helped to raise the tide on all social indicators, including PISA performance. 2 Education Quality and Opportunities for Skills Development in Albania In the context of a rapidly expanding education system, it is remarkable that Albania managed to improve its average PISA scores. It is even more notable that the improvement came from significant advancements made by low-achieving students. An analysis of PISA 2000-2012 results 3 The share of students achieving at least a basic level of proficiency increased moderately from 2000-2012. The performance of low achievers has improved since 2000, resulting in an upturn in the share of students reaching a minimum level of proficiency. In reading, the share of students performing below the “functional literacy” line decreased from 57 percent to 52 percent between 2000 and 2012, and from 64 percent to 61 percent in math during the same period. However, this still means that the majority of students are not able to understand or analyze basic texts or solve simple mathematical problems. Despite the upward trend, Albania’s overall performance remains low. Albania’s PISA 2012 scores are the lowest in all of Europe and Central Asia (ECA) and among the lowest of all PISA-participating countries. Overall, Albania lags around one year of schooling behind countries like Bulgaria, Romania, and Serbia, and around two-and-a-half years of schooling (around 100 PISA points) behind the OECD average. Education quality was not only sustained but improved during Albania’s dramatic expansion of the education system in recent years, but the analysis shows that young people joining the labor market in the immediate future may not possess adequate skills to meet the demands of a modern and knowledge-based economy. The analysis of determinants of PISA 2012 scores was hampered by unreliable student-level data, which limited the analysis to PISA 2009 data. According to OECD, the adjudication results concluded that the PISA Technical Standards were fully met in all countries and economies that participated in PISA 2012 with the exception of Albania. For example, Albania submitted parental occupation data that were incomplete and appeared inaccurate, since there was over-use of a narrow range of occupations. It was not possible to resolve these issues during the course of data cleaning, and as a result, neither parental occupation data nor any indices that depend on these data (such as the Index of Economic, Social, and Cultural Status, ESCS) were included in the international dataset. Due to the low validity of students’ background data in PISA 2012, this study relies mostly on the 2009 PISA survey for the analysis of household characteristics. Significant inequities are observed between student groups, but none of the student groups perform well, leading to the conclusion that education quality is poor across the country. There is evidence of large disparities between groups of different socioeconomic status, geographical location, and gender (e.g., girls substantially outperform their male counterparts). However, none of the groups compare positively to OECD standards: for example, an Albanian student in the 80th percentile of performance in PISA reading scores would only be placed in the 45th percentile among students from OECD countries. Although students in private schools fare considerably better than their peers in public schools, the gap is fully explained by differences in socioeconomic characteristics (which are not subject to modification through policy reform) and access to better school resources (including qualified teachers). Every school can do much more to improve student learning outcomes and to elevate the results from this low base. 4 Education Quality and Opportunities for Skills Development in Albania At the system level, social stratification and preschool education pose important challenges. An in-depth analysis offers two interesting conclusions. First, despite the fact that Albania is relatively homogeneous with respect to residents’ socioeconomic status, the social stratification of schools (the homogeneity of schools in terms of socioeconomic status) is worryingly high, rendering Albania with one of the most stratified educational systems among PISA-participating countries. Second, the analysis finds that although attending preschool in OECD countries has a positive impact on PISA performance (even after controlling for several factors), this effect is nonexistent in Albania. One potential explanation is that the quality of preschools is significantly lower in Albania. An analysis that goes into more detail would be beneficial, but falls beyond the scope of this study. An analysis of PISA math and reading “subscale” skills shows high variation in performance. PISA rotates the in-depth assessment of skills by subject area each time it is administered. For the years that PISA was administered in Albania, 2000 and 2009 focused on reading and 2012 focused on math, and PISA 2015 will focus on science. Compared with the average math and reading performance, results in Albania show higher variation across subscale assessments than in OECD countries. Students performed better in problems related to space and shape and not as well in problems related to data and statistics or change and relationships. In reading, students performed better with more traditional texts than on exercises involving texts contained in sample lists, graphs, or diagrams. Moreover, Albanian students are not good at relating information presented in a text or a mathematical problem to their own personal experiences. Classroom practices show a low quality of teaching and learning strategies in Albania compared to OECD and other ECA countries. PISA 2009 offered the opportunity to assess which types of learning strategies (memorization, elaboration, control, or summarizing) were more commonly used by students and which types of teaching practices (discipline, classroom management, or ability to engage students) were typically used by teachers. The analysis finds that these factors play a significant role in determining learning outcomes, and it also identifies the importance of having teachers who attain at least a tertiary education. The following policy recommendations emerge from this study: Reach national consensus to improve overall education quality. Student achievement as measured 1  by PISA improved from 2000-2012, especially in reading. However, overall performance remains low and the majority of students are below the basic proficiency levels in both reading and math. Despite several reform efforts over the past decade, resource allocation to education remains inadequate, with little hope for increased public expenditures in the context of macro fiscal constraints. In the short term, the government is presented with the challenge of continuing to improve education quality while seeking efficiency gains in the sector. In the medium to long term, it is clear that comprehensive and concerted An analysis of PISA 2000-2012 results 5 reform is required to accelerate progress in the sector, starting with a national consensus to prioritize the enhancement of learning outcomes and ensuring skills development across the board. Without recognition that education reform presents a unique opportunity to produce a cadre of workers with competent and flexible skills, catalytic economic and social development will remain out of reach in Albania. 2  Steer educational resources and qualified teachers to benefit the most disadvantaged students. Some groups systemically lag behind. The most disconcerting trend is the widening gap between students from the highest and lowest quintiles of the OECD ESCS Index from 2000 to 2009, resulting in a performance gap equivalent to about two years of schooling in both math and reading. Furthermore, the gap between boys and girls is more than 60 points in reading, while the urban-rural gap is more than 50 points, equivalent to nearly one-and-a-half years of schooling. Although the entire education sector is resource constrained, more funding should be targeted to benefit boys, students from disadvantaged backgrounds, and those who live in rural areas. Rural schools may need extra school resources to make up for the gap in students’ socioeconomic backgrounds, while incentives should be aligned to ensure that the best teachers are matched with the most difficult students. Related to this, the existence of overlapping catchment areas for schools and the growing number of small schools present opportunities to consider school rezoning options in a way that makes schools less stratified and promotes more equitable outcomes. Strengthen the effectiveness of teacher practices as this is a key driver of quality. The analysis 3  highlights the importance of effective teaching practices, student learning strategies (partially a product of teaching practices), and the positive impact of having a large proportion of qualified teachers in schools. Peer mentoring programs paired with classroom observations, teachers’ professional development focusing on improving classroom effectiveness, and feedback systems for teachers could make the difference in strengthening classroom management techniques and promoting relevant pedagogic techniques. Expand coverage and monitor the quality of preschool education. Preschool education has the 4  potential to mitigate inequities early in life and level the playing field for disadvantaged students. PISA data show that an early start in primary school is strongly correlated with higher reading scores in Albania even after controlling for several factors. Yet an analysis of variance in access to preschool or starting primary education at age six presents disparities along socioeconomic quintiles, as well as between boys and girls. In the context of the government considering the introduction of grade 0 for inclusion in compulsory education, special efforts will be required to ensure that boys and students of low socioeconomic status are targeted for enrollment. Furthermore, the quality of early education should be enhanced and monitored to ensure that the “preschool effect” is attained in Albania. 6 Education Quality and Opportunities for Skills Development in Albania Photo: Jutta Benzenberg Improve the robustness, reliability, and accessibility of data. Albania is the only country whose 5  PISA 2012 household data are not reliable, according to the OECD. As Albania prepares to participate in PISA 2015, including piloting instruments prior to spring 2015, mechanisms to ensure the quality of data collection are critical for evidence-based policy making to take place. International surveys such as PISA may be costly but they have the potential to help policy makers identify weaknesses in the education sector and thus should be exploited to their fullest. An analysis of PISA 2000-2012 results 7 Why is PISA 1. important? An Overview of Albania’s performance on PISA 8 Education Quality and Opportunities for Skills Development in Albania Education and skills are critical for the development of both countries and individuals. International evidence suggests that quality of education is one of the most important determinants of long-term economic growth.2 Hanushek and Woessmann (2007 and 2012) looked at a wide range of student assessment surveys from 1960 onward, including the Trends in International Mathematics and Science Study (TIMSS), the Programme for International Student Assessment (PISA), and the Progress in International Reading Literacy Study (PIRLS). They estimated that an improvement of 50 points in PISA scores would imply an increase of 1 percentage point in the annual growth rate of GDP per capita.3 Top-quality education systems are also associated with democratic governments. Beyond economic growth, education improves the living standards of individuals, as the more educated are able to acquire more and higher-order skills, making them more productive and employable and extending their labor market participation over their lifetime, which in turn leads to higher earnings and better quality of life. Formal schooling also contributes to development of socio-emotional skills like attention, motivation, self- confidence, and physical and emotional health, all important determinants of socioeconomic mobility. Individuals equipped with more education and skills are better prepared to become civically engaged, improve the democratic capital of their country, and create and make use of opportunities. Education is a key ingredient for reducing inequality and increasing shared prosperity. The analysis of detailed data is critical for understanding the determinants of education quality and can play an important role in shaping effective evidence-based education policy. The PISA database is a great resource in the pursuit of this analysis. PISA is a tool for measuring education quality across countries. Introduced in 2000 by the Organisation for Economic Co-operation and Development (OECD), PISA is a worldwide study of 15-year-old school students’ performance on three different disciplines: mathematics, science, and reading. PISA focuses on the competence of students and their ability to tackle real-life problems in those three disciplines and emphasizes skills that are critical for individuals’ personal and professional development. PISA only assesses students who are in the education system, making it the most realistic snapshot available of a country’s education system. However, if dropout rates are high, the results may not be representative of a country’s cohort of 15-year-olds. PISA’s scoring system is standardized so that the mean score for each discipline among OECD countries in year 2000 is 500 points, with a standard deviation of 100 points. According to OECD, 40 points in PISA is equivalent to what students learn in one year of schooling.4 Albania’s education system (see Box 1) was assessed in the PISA rounds of 2000, 2009, and 2012. Albania’s participation in PISA allows us to benchmark it with other countries, measure the extent to which the country has succeeded in promoting education quality, and gauge whether system inequities have been reduced over time. The World Bank’s PISA Country Series seeks to explore the wealth of information collected by PISA to analyze education systems and provide policy makers with options for evidence-based policy reform. Due to its focus on policy, the Country Series aims to address key challenges in Europe and Central Asia (ECA) countries, with a focus on improving education quality and equity. The analysis in An analysis of PISA 2000-2012 results 9 Figure 1 PISA scores and GDP per capita 600 East Asia Singapore Chinese Taipei Hong Kong Korea Japan Switzerland 550 Estonia Finland Netherlands Europe & Poland PISA score in mathematics, 2012 Canada Central Asia Czech Republic Denmark Germany Australia Slovenia Latvia Portugal SpainFrance United Kingdom Norway 500 Russia Slovak Republic Croatia Lithuania Sweden United States Hungary Israel Serbia Turkey Greece Western Europe 450 Romania Bulgaria Cyprus & US/Canada Kazakhstan Thailand Montenegro Mexico Uraguay 400 Costa Rica Albania Argentina Latin American Brazil Colombia Peru 350 0 10000 20000 30000 40000 50000 60000 70000 GDP per capita (PPP), latest (IMF) Source: PISA 2012 and UNESCO 2012. Note: The curve represents a logarithmic approximation of the scatter plot. this report includes a descriptive section in which par with expectations when considering its level of the performance of different subpopulations is public expenditure on education per student. It is discussed, followed by a section that identifies the important, however, to note that Albania’s public determinants of PISA results. The latter section expenditures on education are as low as 3 percent is divided into discussions of the roles of: (i) of GDP, significantly less than that of countries socioeconomic and family background factors; and with similar demographic structure and level of (ii) factors that are heavily influenced by policy, economic development. However, while a certain including tracking, the availability of educational level of financial resources is important to ensure resources, teacher effectiveness (observed through a minimum standard of quality, higher levels of teacher practices and learning strategies), school expenditures and development do not necessarily autonomy and accountability, and pre-primary produce better learning outcomes. In lower-middle- education.5 The standard framework of the Country income countries like Albania, more investment can Series is flexible enough to allow each report to help improve quality but additional policy efforts reflect the country-specific context. are needed to realize efficiency gains and achieve sustainable improvements in learning outcomes. Albania’s PISA performance is slightly below what should be expected given its current level Albania’s performance on PISA 2012 represents of economic development (Figure 1). a slight improvement from 2009 in reading For example, comparator countries such as Thailand and science and a substantial leap in math and Serbia performed better on PISA 2012 math, (Table 1). Since 2000, Albanian students have made despite having similar levels of GDP per capita. significant progress in reading (especially between On the other hand, Albania’s performance is on 2000 and 2009), partly because their performance 10 Education Quality and Opportunities for Skills Development in Albania Box 1 Albania’s Education System in this discipline was lower than that in math and Albania has a population of 2.8 million people, the science in the first round. Its math and science vast majority of whom are of Albanian ethnicity. performance has gone up, but not at the same pace Other ethnic groups include Romas, Greeks, and as seen in reading. While the trend shows positive Macedonians. The education system currently developments since 2000, Albania started from serves approximately 630,000 students in pre- a very low base and still has tremendous room for university education (including preschool). improvement. The school-age population is declining in Albania: in 2007, there were 700,000 pre-university The improvements between 2000 and 2012 students. Moreover, according to UN estimates, were realized despite substantial increases in the population of those aged 19 and younger enrollment and the attendant changes in the is projected to shrink from 32 percent of the total PISA sample composition. In 2000, prior to the population in 2010 to 24 percent in 2025. reorganization of the structure of Albania’s education system, 80 percent of students participating in PISA Albania’s education system consists of: (i) were enrolled in upper secondary education. At that preschool for children aged 3-6, which is free but time, the gross enrollment rate at that education not compulsory; (ii) compulsory basic education level was only 40 percent,6 which implies that many for children aged 6–15, which has two cycles—a 15-year-olds were out of school and thus excluded lower cycle consisting of grades 1–5 and an upper from PISA sampling. Because of selection biases,7 cycle consisting of grades 6–9 ; (iii) general upper it is likely that if the out-of-school children had been secondary education (lasting three years) and two part of the PISA sample, the average performance levels of vocational education (lasting three and would have been substantially lower. In 2009 and five years), with all graduates eligible to take 2012, the share of students in upper and lower the exam for admission into higher education; secondary was more evenly distributed (50 percent in (iv) higher education, which offers short-cycle upper secondary and 42 percent in lower secondary professional studies lasting two years; and (v) in 2012) and the gross enrollment rate in upper three-year bachelor programs, two-year master’s secondary had risen markedly, to 72 percent in 2009 programs, and doctorate programs that last three and 83 percent in 2012. Combined with the fact that or more years. Education is compulsory until the enrollment rates in lower secondary have historically end of grade 9. Finally, at the end of upper been above 90 percent, this means that substantially secondary education, pupils sit for the state more 15-year-olds were captured in the PISA Matura exam, the results of which are used both sampling in 2009 and 2012. In this scenario of access as an exit exam from upper secondary and as an expansion, it is remarkable that Albania managed to entrance exam for university admission. All levels improve scores. of education are offered by both public and private institutions. Source: World Bank (forthcoming) An analysis of PISA 2000-2012 results 11 Table 1 Albania’s PISA performance by discipline, 2000-2012 2000 2009 2012 Reading 349 385 (+36*) 394 (+9) Math 381 377 (-4) 394 (+17*) Science 376 391 (+15*) 397 (+6) Source: PISA 2000, 2009, and 2012. Note: Change in score from previous round is in parentheses, with *indicating statistical significance. Albania’s PISA score improvements coincide The share of students below basic proficiency with the launch of intensive reform efforts in levels has decreased moderately since 2000. its education sector. The government initiated PISA categorizes scores in six levels of proficiency; the education sector strategy formulation process students who score below level 2 in the reading and in 2002, which resulted in the 2004 passage of math tests are considered functionally illiterate the National Education Strategy (NES) and the and innumerate, respectively.8 According to the first attempt to develop a long-term roadmap 2012 data, 60 percent of 15-year-old students score for the sector. The NES served as the catalyst for below level 2 in math (Figure 2a), meaning that they a range of reforms, including improved teacher are not able to understand and solve simple math recruitment, compensation, and management; problems, severely limiting their development and a revised curriculum for basic and general upper skill acquisition process. The long-term trend is a secondary education; enhanced transparency and source of concern, with the share of students scoring accountability through reform of the Matura, the below level 2 fluctuating from 64 percent in 2000 to national student assessment; reduced price and 68 percent in 2009, before dipping in 2012. Based on improved textbook quality through a reformed PISA reading scores, about 52 percent of Albanian procurement process; and provision of textbook students are considered functionally illiterate, but subsidies to the poorest households. While it is there has been substantial improvement since 2000, beyond the scope of this report to determine a causal when 70 percent of students were below level 2 in relationship between the range of policy reforms reading. Figure 2c depicts the evolution of math and improved PISA scores, this report shows that scores between 2000 and 2012 and illustrates that socioeconomic variables played an important role. more students are pushing toward the functional The explanatory analyses below show that the effect literacy threshold. However, in some comparator of education variables cannot be disentangled from countries, Romania and Bulgaria in particular, the the effect of socioeconomic variables. Furthermore, share of students below level 2 fell a dramatic 10-16 with economic growth averaging 4.9 percent per percentage points between 2006 and 2012 in both year from 2000-2012, a viable explanation is that the math and reading (Figure 2b). general improvement in economic conditions helped to raise the tide on all social indicators, including PISA performance. 12 Education Quality and Opportunities for Skills Development in Albania Figure 2 Distribution of students by proficiency level in math: (a) Albania’s progress from 2000-2012; (b) Albania and comparators in 2012; (c) Evolution of math scores in Albania from 2000-2012 80% 100% 70% 80% 60% n 2000 n 2009 n 2012 60% 50% 40% 40% 30% 20% 20% 0% Albania Montenegro Bulgaria Romania Serbia Greece ECA-EU11 OECO 10% 0% Below Level 2 Level 3 Level 4 Level 5+ n Level 5+ n Level 4 n Level 3 n Level 2 n Below 2 Level 2 0.006 ––– 2000 ––– 2012 0.005 ––– Functional literacy 0.004 threshold Frequency 0.003 0.002 0.001 0 0 100 200 300 400 500 600 700 800 PISA scores Source: PISA 2000, 2009, and 2012. Despite improvements, Albania’s performance approximately one year of schooling. While some is at the bottom among regional comparator of the comparators have not improved their results countries (Figure 3). Albania’s scores are the lowest substantially between 2000 and 2012, and Albania is among ECA countries in math and science and the the poorest in this group, the gap that Albania must second lowest in reading. A composite score of the breach remains vast. three subjects shows that Albania lags around 15-30 PISA points behind Montenegro, around 40 PISA points behind Bulgaria, and over 100 points behind the OECD, with each 40 point gap equivalent to An analysis of PISA 2000-2012 results 13 Figure 3 PISA 2012 scores for Albania and comparator countries versus ECA and OECD averages 510 490 470 450 430 410 390 370 Greece Greece Albania Bulgaria Romania Serbia ECA-EU11 Montenegro Greece Montenegro Romania Serbia Bulgaria OECD Montenegro OECD Albania Bulgaria Romania Serbia ECA-EU11 OECD Albania ECA-EU11 Source: PISA 2012. Math Reading Science 25x3=75 14 Education Quality and Opportunities for Skills Development in Albania An analysis of PISA 2000-2012 results 15 2.Poor quality inequalities and persistent 16 Education Quality and Opportunities for Skills Development in Albania The determinants and drivers of education quality are analyzed in three dimensions: (i) individual characteristics; (ii) system-level policies; and (iii) school-level policies. PISA scores are first used as a measure of education quality, and then the effects of variables in the PISA student and school questionnaires on learning outcomes are analyzed. We employ different analytical techniques, and broadly divide variables into individual and school characteristics, with subgroups of variables within school characteristics (peer characteristics, school resources and practices, and system variables). The unreliability of PISA 2012 data limits the analysis mainly to results from PISA 2009 and 2000 (see Box 2). Despite advances, the overall low student performance is likely to have a detrimental impact on long-term economic development and opportunities for intergenerational prosperity. The quality of education across all groups is a critical concern in Albania. As noted above, the majority of students have performed below basic proficiency levels since 2000. Furthermore, a top student in reading in Albania – one who is above 80 percent of all Albanian students – performs below 55 percent of OECD students, making her a mediocre achiever in international comparison (Figure 4).9 The picture is similar for math and indicates that the top students in Albania are still below most students in the OECD.10 The poor skills acquisition demonstrated through the PISA results presents a crisis in the making for Albania’s economic and social development prospects. The next section elaborates on the inequities in learning outcomes in the education system, to identify the groups most at risk. An analysis of PISA 2000-2012 results 17 Box 2 PISA’s Index of Economic, Social, and Cultural Status and PISA Data Validity in Albania Created by OECD, PISA’s Index of Economic, Social, and Cultural Status (ESCS) is a multidimensional measurement that takes into account information reported by students on their family’s wealth and occupational, educational, and cultural background. It is derived from a combination of three other indices: (i) an index of the highest occupational status of parents, indicating not only labor market status, but also the type of job held by parents; (ii) an index based on the highest level of parental education in years of schooling; and (iii) an index of family home possessions, which itself consists of a combination of the family’s possessions (such as cars, bathrooms, or technological devices) and educational resources (such as desks, computers, textbooks, and the number of other books), as well as the type of cultural possessions (such as the type and genre of books or the works of art). The ESCS Index is the most important determinant of student achievement and is therefore crucial for analysis of the quality of education. According to OECD, the adjudication results concluded that the PISA Technical Standards were fully met in all countries and economies that participated in PISA 2012 with the exception of Albania. For example, Albania submitted parental occupation data that were incomplete and appeared inaccurate, since there was over-use of a narrow range of occupations. It was not possible to resolve these issues during the course of data cleaning, and as a result, neither parental occupation data nor any indices that depend on these data (such as the ESCS Index) are included in the international dataset. Due to the low validity of student background data in PISA 2012, this study relies mostly on the 2009 PISA survey for the analysis of household characteristics. Source: OECD 2014. Instead of catching up with the country’s average, students from disadvantaged backgrounds are lagging farther behind. This has serious implications for the country’s development and for social cohesion, as education is a key mechanism for intergenerational mobility. 18 Education Quality and Opportunities for Skills Development in Albania Figure 4 Distribution of PISA 2012 reading scores in Albania and the OECD 0.005 20th percentile 80th percentile in 0.004 in Albania Albania would be would be the the 45th percentile in the OECD 3rd percentile 0.003 in the OECD Frequency ––– Albania ––– OECD 0.002 ––– 313 points ––– 485 points 0.001 0 100 200 300 400 500 600 700 800 700 800 PISA reading score (2012) Source: PISA 2012. Inequities in Albania: Which students lag behind? The overall equality of opportunities in Albania resources, preschool attendance, and better teaching is comparable to that of other countries in practices in the classroom.12 While high in absolute the region. The index of equality of opportunities terms, this difference is similar to that found in (Figure 5) shows the importance of individual countries such as Serbia, Croatia, and Montenegro. characteristics (gender, age, and socioeconomic However, without sustained improvements for all, status) in determining student performance. disadvantaged students are unlikely to increase their In Albania, using 2009 PISA data, individual future living standards. While average score growth characteristics explained around 22 percent of the is important, it is also crucial to foster improvements differences in reading achievement, which places among the bottom 40 percent of a country’s student the country around the ECA average.11 population.13 Between 2000 and 2009, students in the bottom 40 percent in terms of socioeconomic Performance trends show a widening gap status made advancements in reading at a level on between the average student and students par with the country’s average (Figure 6). In math, from disadvantaged backgrounds between the performance of the bottom 40 percent of students 2000 and 2009. Although Albania has relatively low fell more than the average did. These results present income inequality, there is a significant gap between a gloomy picture: instead of catching up with the the performance of students in the highest and lowest country’s average, students from disadvantaged quintiles of the ESCS Index (see Table 2): 93 points in backgrounds are lagging farther behind. This has reading and 78 points in math, equivalent to about serious implications for the country’s development two years of schooling in both cases. The gap also and for social cohesion, as education is a key reflects differences in access to high-quality school mechanism for intergenerational mobility. An analysis of PISA 2000-2012 results 19 Figure 5 Index of equality of opportunities: Albania and other ECA countries, 2009 0.35 0.30 More equality of opportunities 0.25 0.20 0.15 0.10 0.05 Estonia Kazakhstan Russia Serbia Montenegro Croatia Czech Republic Poland Romania Latvia Albania Slovak Republic Slovenia Turkey Lithuania Bulgaria Hungary Source: Authors’ calculations based on PISA 2009 . Note: The index is the percent of the variance in reading scores explained by the main predetermined characteristics (age, gender, and socioeconomic status) in a linear regression (Ferreira and Gignoux 2011). 2+3+3=8 abcdefghijkl 20 Education Quality and Opportunities for Skills Development in Albania Table 2 Difference in PISA 2009 performance between top and bottom socioeconomic quintile groups in Albania Reading Math Top Bottom Top Bottom 20% 20% Difference 20% 20% Difference Albania 439 346 93 421 343 78 Bulgaria 502 360 142 496 369 128 Greece 534 436 98 517 423 94 Croatia 517 434 83 502 420 82 Montenegro 451 364 87 443 360 83 Romania 472 375 97 468 384 84 Serbia 486 408 78 493 402 91 Source: Authors’ calculations based on PISA 2009. Figure 6 Albanian students’ performance on math and reading in PISA 2000-2009 by socioeconomic group 390 385 380 381 377 PISA Performance Points 370 364 365 360 357 350 349 340 330 332 2000 2009 2000 2009 Reading Maths Average Bottom ESCS 40% Source: PISA 2000 and 2009. An analysis of PISA 2000-2012 results 21 The significant gender gap in reading poses an important challenge for Albania; its education system is failing to provide quality education to all of its students, but it is particularly failing boys. Gender disparities Albanian girls outperform boys by over 60 is only 2.5 points. Moreover, while 78 percent of points in reading, equivalent to 1.5 years of OECD students start formal basic education at the schooling. In most countries, girls fare better than age of six or younger, only 49 percent of Albanian boys in reading and tend to perform at equal levels in students did so in 2009. The gender gap in Albania math. In 2009, the gender gap in reading in Albania is also prominent in this dimension: only 44 percent was the highest among all comparators in the region of boys start education at six or younger versus 54 (Figure 7). The large difference in performance percent of girls. In OECD countries, there is no such between girls and boys in reading has remained gap. Most importantly, PISA data show that an early fairly stable while the gap in math has fallen, from 20 start is strongly correlated with higher reading points in 2000 to 10 points in 2009. The significant scores in Albania, even after controlling for several gender gap in reading poses an important challenge factors, including socioeconomic status, age, grade, for Albania; its education system is failing to provide gender, teacher practices, learning strategies, and quality education to all of its students, but it is quality of educational resources. Starting primary particularly failing boys. The average OECD student education one year early accounts for an increase of scores approximately 140 points above the average approximately 9 points in PISA reading scores.14 Albanian boy, a difference of approximately 3.5 years of schooling. Urban-rural disparities The gender gap emerges in students’ access to The urban-rural gap is substantially higher than and age of starting preschool. Only 48 percent of that found in other countries. In 2009, the urban- Albanian boys have more than one year of preschool rural gap in reading was 54 PISA points, almost 1.5 while 58 percent of girls do. In OECD countries, years of schooling. In math and science, the gaps the share of children who attended more than one were slightly less severe, but not less worrying: 43 year of preschool is 70 percent and the gender gap and 39 points, respectively, equivalent to roughly one year of schooling.15 The average urban-rural gaps 22 Education Quality and Opportunities for Skills Development in Albania Figure 7 Gender gap in reading, PISA 2009, selected countries 65 60 55 50 45 40 35 Italy Montenegro Serbia Romania Turkey Greece Croatia Slovakia Slovenia Albania Bulgaria Source: PISA 2009. among ECA countries are 38 points in reading and difference in reading outcomes between students in 32 points in math, which are high but significantly private and public schools is equal to 64 PISA points lower than in Albania. Similar to the socioeconomic (equivalent to one-and-a-half years of schooling), and status gap, the urban-rural gap reflects differences is fully explained by socioeconomic characteristics, in access to better quality schools. The urban-rural availability of educational resources in school, and gap presents a critical challenge for Albania, as over the proportion of qualified teachers (Table 3).16 45 percent of its PISA sample live in rural areas, substantially higher than the 30 percent average The Albanian diaspora across all participant countries and the 37 percent average of ECA countries. Hence, Albania has a In Greece and Switzerland, the Albanian higher share of its population achieving very poordiaspora gets slightly better results than results. The change from 2009 to 2012 cannot be students in Albania, but this is not the case in analyzed because of the unreliable data in the latter Montenegro. At 29 percent, Albania’s emigration PISA round. rate is among the highest in Europe.17 The highly educated follow the same trend, with an emigration Public and private schools rate of 27 percent. The trend was particularly acute in the 2000s, when the overall emigration rate rose The gap between students in private and public by more than 9 percentage points. PISA data allow schools is largely explained by differences the identification of children who reside in another in household characteristics and school country but were either born in Albania or have at resources. In the PISA 2009 sample, 11 percent of least one parent born in Albania. Figure 8 shows students in Albania attended private schools. The how the Albanian diaspora fares in Switzerland, An analysis of PISA 2000-2012 results 23 Table 3 Characteristics of student by school type in 2009 Public Schools Private Schools Average reading score (PISA 2009 score) 378 442 Father working full-time (%) 56.0 72.2 Mother working full-time (%) 27.7 46.4 Highest parental education (years) 12.1 13.5 Teachers with higher education (%) 77.5 95.2 Index of Quality of Educational Resources -0.92 0.69 Source: PISA 2009. Note: Index of teaching practices and learning strategies are normalized at the OECD level. Index of socioeconomic has a full-sample average of -0.55 and a standard deviation of 1.21. The index of quality of educational resources has a full-sample average of -0.28 and a standard deviation of 1.21. Figure 8. The PISA 2009 performance of the Albanian diaspora 550 Country Country Average Average 500 Country 450 Average 400 350 300 Albania Greece Montenegro Albania Greece Montenegro Albania Greece Montenegro Switzerland Switzerland Switzerland Math Reading Science Source: PISA 2012. 24 Education Quality and Opportunities for Skills Development in Albania Greece, and Montenegro, the countries with the Social stratification in Albanian schools is largest number of recent Albanian immigrants. In high compared to other countries (Figure 9). all three countries, Albanians perform below the We define the Index of School Social Stratification host country’s average. However, in Switzerland as the correlation between the PISA individual and Greece, despite scoring substantially below socioeconomic status and the average socioeconomic the host country average, immigrants still perform status at the school level.20 In a world without social above the average of Albania, while Albanians in stratification (thus an index equal to zero), families Montenegro fare worse than the Albanian average. from different socioeconomic backgrounds would The socioeconomic status of students in Albania randomly settle across the country and students from does not seem to be different from that of immigrant different backgrounds would study together, making students in Greece or Montenegro. However, the schools more diverse. However, households tend to socioeconomic status of Albanian immigrants tends co-locate in neighborhoods with other households to be slightly higher in Switzerland than that of similar to them, and students tend to attend school students in Albania. with peers who have similar socioeconomic status as a result of spatial inequalities. It should be noted that The importance of learning from peers the best performing countries in PISA tend to have less stratified education systems. The peer effect is relatively strong in Albania, which is an important consideration in the Even though large socioeconomic disparities context of a declining student population. cannot be resolved in the short term, policies The previous analyses examined the importance can address school stratification. As seen above, of individual characteristics, which explain 22 the peer interaction that takes place in school plays percent of score differences in reading, about the a very important role in determining children’s same as ECA’s average of 23 percent. Individual achievement. In Albania, despite relatively low characteristics averaged at the school level (i.e., peer income inequality at the national level, students characteristics) also explain part of the differences from similar backgrounds tend to cluster into the in reading scores.19 Jointly with individual same schools. This may be the result of an explicit characteristics, peer effects explain 29 percent of the choice of parents who prefer their children to be variation in reading scores in 2009. Although this is surrounded by children of similar socioeconomic not high compared to other countries in the region, backgrounds, but may also happen as a consequence it suggests that students’ performance is influenced of other factors, such as the location in which each by their peers. Data from a recent school mapping group lives or an exam that streams students into exercise, although unofficial, suggest that there is a different tracks. The school network in Albania is network of small schools in Albania. This peer effect such that 69 percent of the schools have two or more on student learning should be taken into account as other schools in the same catchment area, while 13 the government explores options for rezoning schools percent have one other school. Even in rural areas, 68 in the context of a declining student population. percent of the schools have at least one other school An analysis of PISA 2000-2012 results 25 Figure 9 Index of School Social Stratification in PISA-participating countries, 2009 0.65 0.60 0.55 OECD Average 0.50 0.45 0.40 0.35 0.30 Norway Estonia Kazakhstan Italy Azerbaijan Finland Switzerland Montenegro Canada Ireland Greece Turkey Denmark Great Britain Croatia Slovak Republic Serbia Latvia Romania Russian Federation Kyrgyc Rep. Poland Czech Republic Slovenia Georgia Albania Bulgaria Source: Authors’ calculations based on PISA 2009. Note: The index goes from 0 to 1. A higher index indicates a higher correlation between student and school socioeconomic status. The figure includes a selected number of PISA-participating countries. in the same area. This provides policy makers with an opportunity to promote social integration and Cognitive skills consequently improve education in most parts of the country. International evidence shows that systems that stratify students according to socioeconomic gaps start status increase the inequality of student performance without improving average performance levels.21 emerging early Early childhood policies and inequalities A “preschool effect” on student learning, observed in OECD countries, is not present in in access to Albania. The reading performance gap between students who did not attend preschool and those who preschool attended more than a year is 32 points (and 14 points comparing no preschool with one year of preschool), perpetuate significantly below the gap in OECD countries (66 points) but not far from regional comparators such learning gaps. as Croatia (28 points) and Slovenia (40 points). After controlling for other factors (such as socioeconomic status, age, grade, and gender), the positive impact 26 Education Quality and Opportunities for Skills Development in Albania of having attended more than a year of preschool in-depth assessment with a larger set of questions. falls to 5 points, which implies that part of the effect The detailed assessment was on reading in 2000 and of attending preschool is mixed with the effects 2009, on math in 2003 and 2012, and on science in of other characteristics. Although the data do not 2006. The 2015 round will focus again on science. allow for a clear-cut answer as to why the effect of Thus for Albania, two detailed assessments on preschool is not as strong in Albania, one possible reading and one detailed assessment of math explanation lies in the quality of programs provided. are available. Analysis beyond of the scope of this study is needed to answer this question, but global evidence shows that Math skills in PISA 2012 providing quality preschool education is important for promoting children’s social, emotional, physical, The PISA math 2012 subscale assessment and cognitive development; it also increases school measured individuals’ ability to formulate, readiness, which helps learning (Heckman and employ, and interpret mathematics in a LaFontaine 2010; Heckman 2008; Engle et al. variety of contexts and content areas. In PISA, 2011). Cognitive skills gaps start emerging early the concept of mathematical literacy includes: (i) and inequalities in access to preschool perpetuate mathematical reasoning; (ii) usage of mathematical learning gaps. In addition to quality issues, Albania concepts, procedures, and facts; (iii) tools to describe, also faces a challenge with access and equity. The explain, and predict phenomena; and (iv) the role Living Standards Measurement Survey (LSMS) that mathematics plays in the world and the need to 2012 data allow for disaggregation by consumption make well-founded judgments and decisions needed quintiles, and show enrollment rates varying by constructive, engaged, and reflective citizens. between 20 percent (for the poorest) and 48 percent Furthermore, mathematic literacy as defined by (for the richest).22 PISA not as an attribute that an individual has or does not have; rather, it can be acquired to a greater or How Do Classroom-Level lesser extent, and it is required in varying degrees in society. PISA seeks to measure not just the extent to Practices Affect Learning? which students can reproduce mathematical content knowledge, but also how well they can extrapolate PISA offers the opportunity to fully explore one from what they know and apply their knowledge of subject area every three years, even though mathematics in new situations. The questions faced all three subjects are assessed every time PISA by students are framed in four real-world context is administered. PISA seeks to assess not merely categories: Personal, Societal, Occupational, and whether students can reproduce knowledge, but Scientific. Box 3 illustrates a typical math question also to examine how well they can extrapolate from from PISA 2012. what they have learned and apply it in unfamiliar settings, both in and outside of school. The detailed test of “subscale” skills of a given subject area is an An analysis of PISA 2000-2012 results 27 Box 3 Sample PISA 2012 Math Problem: Which Car? The following problem was part of PISA 2012’s main survey. The first question requires interpretation and falls in the “uncertainty and data” content category while the other two questions measure “quantity” and require students to employ mathematical facts. The problem’s difficulty level is rated at 327.8 points for question 1, 490.9 points for question 2, and 552.6 for question 3. In Albania, 80 percent of students were able to reach the level of difficulty of question 1 (on the math average). However, this figure dropped dramatically to only 13 percent for question 2 and less than 3 percent for question 3. In contrast, in OECD countries, 96 percent of students solved questions at the level of difficulty of question 1, 47 percent for question 2, and 25 percent for question 3. Source: OECD 2014. 28 Education Quality and Opportunities for Skills Development in Albania Figure 10 Math contents and process categories in PISA ç problem in context ç formulate mathematical problem ç employ quantity uncertainty and data change and relationships evaluate space and shape results in context Source: OECD 2014. ç The PISA math framework is a sophisticated tool for connecting student’s mastery of interpret mathematical results context. This process involves four skills defined by PISA as “processes,” assessed in 2012 as: formulate mathematical processes and contents. The a mathematical situation according to the concepts math subscale assessment evaluates capacity and relationships identified, employ mathematical in four content categories (Figure 10): quantity facts, procedures, and reasoning to obtain results (incorporates the quantification of attributes of (usually involving calculation, manipulation, and objects, relationships, situations, and entities), computation), interpret the results in terms of the uncertainty and data (understanding messages original problem to obtain the “results in context,” embedded in data, and appreciating the variability and finally, evaluate the outcomes and their inherent in many real processes), change and reasonableness in the context of the problem.23 relationships (temporary and permanent relations among objects and circumstances), and space Albanian students perform significantly and shape (phenomena encountered in patterns, better in content related to space but have object properties, positions, representations, visual difficulties relating math problems to real- information, navigation, and dynamic interactions). life situations (Figure 11). Compared with the Figure 10 also shows a schematic of the stages faced average math performance,24 Albania’s results by a student when solving a real-life problem through show a high variation across subscale assessments the mathematical modelling cycle. The action compared with OECD countries. Relative to overall begins with identifying the problem in context and performance, students in Albania perform well finishes when the results of the problem are found in problems related to space and shape, usually in a context and again are reflected in the problem related to geometry, but underperform in subscales An analysis of PISA 2000-2012 results 29 Figure 11 PISA 2012 performance on different math subscales compared to each group’s average performance 25 Performance difference between each content/process 20 Interpreting/ Evaluating subscale and the average mathematics scale 15 Space and Uncertainty shape and Data Employing 10 Quantity Formulating Change and relationships 5 0 -5 -10 Contents Processes -15 n OECD n Albania Source: PISA 2012. Note: Albania’s average score is 394 and OECD’s average score is 494 of change and relationships, uncertainty and data, The PISA reading framework is built on three and quantity. Results in processes subscales show major characteristics: texts, aspects, and that students underperform when they need to use situations. Although texts are differentiated in contextualization skills to relate the problems to different characteristics (medium, environment, real-life and everyday situations. type, and format), performance on text format is the only one reported in PISA through two Reading skills in PISA 2009 formats: continuous texts (sentences organized into paragraphs, which may fit into even larger structures) The PISA 2009 subscale assessment of reading and non-continuous texts (smaller sentences, usually skills measured students’ ability to actively, in sample lists, graphs, diagrams, or catalogues), purposefully, and functionally apply reading in although there are also mixed and multiple texts. a range of situations. PISA defines reading literacy Aspects are measured as PISA reading subscales as understanding, using, reflecting on and engaging with three categories: access and retrieve (skills with written texts to achieve one’s goals, to develop associated with finding, selecting, and collecting one’s knowledge and potential, and to participate in information), integrate and interpret (which involves society. Understanding refers to the reader’s ability understanding the relationships between different in constructing meaning from text; using refers to parts of a text or making meaning from something the kind of reading that is directed toward applying not stated in the text), and reflect and evaluate information in a text to an immediate task; and (which involves drawing on knowledge, ideas, or reflecting on means that readers relate what they are values external to the text). Finally, situations intend reading with their thoughts and experiences. to maximize the diversity of content included in 30 Education Quality and Opportunities for Skills Development in Albania Box 4 Sample PISA 2009 Reading Question: Labor The following question was part of PISA 2009’s reading assessment. It is an example of non-continuous text; the subscale skill measured is the capacity to retrieve information. The question asked was: “How many people of working age were not in the labor force? (Write the number of people, not the percentage.)” Partial credit was given to students who answered “949.9” and full credit was given to those who were able to figure out that the actual number of people is 949,000. The level of difficulty of the partial credit answer was rated at 485, and 631 for the full credit. Only 20 percent of Albanian students achieved the level of difficulty of the partial credit question (on the reading average). Less than 1 percent of Albanian students reached the level of the full credit question. In contrast, in OECD countries, 65 percent of students solved questions at the level of difficulty of the partial credit answer, while 28 percent of students solved the questions at the level of difficulty of the full credit answer. Source: OECD 2014. An analysis of PISA 2000-2012 results 31 Figure 12 PISA 2012 performance on different reading subscales compared to each group’s average reading performance Continuous Non- Access and Integrate and Reflect and Texts Continuous retrieve Interpret Evaluate 10 Texts Performance difference between each content/process 5 subscale and the average mathematics scale 0 -5 -10 -15 Texts Aspects -20 n OECD n Albania Source: PISA 2009.Note: Albania’s average score is 385 and OECD’s average score is 493. the PISA reading survey; for example, personal, Teaching practices and learning public, educational, and occupational situations are strategies in reading represented. Teaching practices and learning strategies Albanian students have a much better have a major impact on performance.26 understanding of continuous text than of non- To understand the extent to which poor learning continuous text, and there is a need to improve techniques in the classroom hamper student their reflection and evaluation skills. Comparing development, this study creates two indices the reading subscale results with the average reading that account for: (i) the control, elaboration, performance,25 Albania shows much more variation memorization, and understanding strategies of across subscales that that seen in OECD countries, students in class; and (ii) how teachers (as reported which means there is large room for improvement by students) engage students, ensure an adequate in some subscales. In particular, students perform disciplinary climate, manage assignments, and help better with more traditional texts than with texts students relate their own knowledge.27 Analysis contained in sample lists, graphs, or diagrams of PISA scores worldwide finds that higher values (more than 25 points of difference, equivalent to of these two indices are needed to ensure learning a half year of schooling). Moreover, PISA reveals for all.28 Furthermore, the proportion of qualified Albanian students’ weaknesses in relating their teachers (those with tertiary education) in a own experiences to the text, reflecting a disconnect school positively affects reading scores: for every between what students learn and their ability to apply additional 10 percentage point increase in the share this knowledge in real-life situations. of a school’s qualified teachers, there is an increase of 3 points in the PISA reading score. The analysis 32 Education Quality and Opportunities for Skills Development in Albania Figure 13 Decomposition of the reading performance gap between villages/towns and cities (PISA 2009) 80 70 60 PISA Performance points 50 n Individual characteristics n Peer characteristics 40 n Learning strategies n Teaching practices 30 n Education resources n Unexplained l Actual gap 20 10 0 Village Small Town Town Source: Authors’ calculations based on PISA 2009 data. Note: Twofold Oaxaca decompositions were conducted comparing each geographical location’s gap with cities, and including the indices of learning strategies and teaching practices. “Unexplained” factors include the aggregation of gaps explained by returns to individual and school characteristics and the residual part of the threefold Oaxaca decomposition.29 accounted for: (i) learning strategies employed by the gap by around 10 points (Figure 13). In the students; (ii) teaching practices used in schools; results shown in Figure 13, “unexplained” refers and (iii) the quality of educational resources; while to the combination of systemic change factors (iv) controlling for several individual and peer and the residual effect not captured by PISA data, characteristics as previously described. including governance and management practices, social and cultural norms, and teacher and principal Spatial disparities are largely explained by effectiveness. individual and peer characteristics, although effective teacher policies could narrow the Learning strategies and teaching practices gap. Analysis of the determinants of school quality explain an important part of the reading shows the critical role of teaching practices and performance gap between girls and boys. The learning strategies for both Albania and other PISA- previous section showed that in most countries, girls participating countries. Decomposing differences outperform boys in reading, usually by a difference in reading performance between students in cities of at least one year of schooling. A decomposition of (with 100,000-1,000,000 people) and smaller the determinants of the reading gender gap features geographical locations shows that differences are learning strategies as a critical reason why boys lag mainly driven by individual and peer socioeconomic behind girls (primary education entrance age and characteristics. However, for less populated locations teaching practices are also important). The largest like villages (less than 3,000 people), small towns part of the gap (80 percent), however, is explained (3,000-15,000 people), and towns (15,000-100,000 by differences in returns to individual and school people), an improvement in effective learning characteristics, which suggests that boys and girls use strategies and teaching practices could help reduce different learning patterns for reading.30 An analysis of PISA 2000-2012 results 33 3. Recommendations 34 Education Quality and Opportunities for Skills Development in Albania An effective assessment system tends to comprise three main types of assessment activities, each of which serves a different purpose and addresses different information needs. These three main types are: classroom assessment, examinations, and large-scale, system-level assessments. In the latter category, the assessments are further split into national and international assessments. PISA is a large-scale, international assessment. In Albania, it offers the only source of information on student learning outcomes that is comparable over time. PISA defines itself as a competency-based assessment that is curriculum-free. Competencies are understood as what students know and can do with their knowledge, and are selected from a pool of tasks a person should be able to master to be successful in life. Although PISA results can be used to inform curricular reforms, assessment programs, and professional development for teachers, they should not be the sole source of information for policy makers. An integrated assessment system comprising the full range of assessments described above should reflect the content of the curriculum, be objective and fair, and provide results in a way that supports the system to identify areas where further research or remedial action are needed. While PISA results present a good starting point to discuss education reform, there are limits to the conclusions that can be drawn from them. With this caveat in the mind, the following policy recommendations emerge from this study: Reach national consensus to improve overall education quality. Student achievement as measured 1  by PISA improved from 2000-2012, especially in reading. However, overall performance remains low and the majority of students are below the basic proficiency levels in both reading and math. Despite several reform efforts over the past decade, resource allocation to education remains inadequate, with little hope for increased public expenditures in the context of macro fiscal constraints. In the short term, the government is presented with the challenge of continuing to improve education quality while seeking efficiency gains in the sector. In the medium to long term, it is clear that comprehensive and concerted reform is required to accelerate progress in the sector, starting with a national consensus to prioritize the enhancement of learning outcomes and ensuring skills development across the board. Without recognition that education reform presents a unique opportunity to produce a cadre of workers with competent and flexible skills, catalytic economic and social development will remain out of reach in Albania. An analysis of PISA 2000-2012 results 35 2  Steer educational resources and qualified systems for teachers could make the difference in teachers to benefit the most disadvantaged strengthening classroom management techniques students. Some groups systemically lag behind. and promoting relevant pedagogic techniques. The most disconcerting trend is the widening gap between students from the highest and lowest 4  Expand coverage and monitor the quality of quintiles of the OECD ESCS Index from 2000 to preschool education. Preschool education has 2009, resulting in a gap of performance equivalent the potential to mitigate inequities early in life and to about two years of schooling in both math and level the playing field for disadvantaged students. reading. Furthermore, the gap between boys and PISA data show that an early start in primary girls in reading is more than 60 points, while the school is strongly correlated with higher reading urban-rural gap is more than 50 points, equivalent scores in Albania even after controlling for several to nearly one-and-a-half years of schooling. factors. Yet an analysis of variance in access to Although the entire education sector is resource preschool or starting primary education at age six constrained, more funding should be targeted presents disparities along socioeconomic quintiles, to benefit boys, students from disadvantaged as well as between boys and girls. In the context backgrounds, and those who live in rural areas. of the government considering the introduction Rural schools may need extra school resources to of grade 0 for inclusion in compulsory education, make up for the gap in students’ socioeconomic special efforts will be required to ensure that boys backgrounds, while incentives should be aligned and low socioeconomic status students are targeted to ensure that the best teachers are matched with for enrollment. Furthermore, the quality of early the most difficult students. Related to this, the education should be enhanced and monitored to existence of overlapping catchment areas for ensure that the “preschool effect” is attained in schools and the growing number of small schools Albania. present opportunities to consider school rezoning options in a way that makes schools less stratified 5  Improve the robustness, reliability, and and promotes more equitable outcomes. accessibility of data. Albania is the only country whose PISA 2012 household data are not reliable, 3  Strengthen the effectiveness of teacher according to the OECD. As Albania prepares practices as this is a key driver of quality. The to participate in PISA 2015, including piloting analysis highlights the importance of effective instruments prior to spring 2015, mechanisms to teaching practices, student learning strategies ensure the quality of data collection are critical (partially a product of teaching practices), and for evidence-based policy making to take place. the positive impact of having a large proportion International surveys such as PISA may be costly of qualified teachers in schools. Peer mentoring but they have the potential to help policy makers programs paired with classroom observations, identify weaknesses in the education sector and teachers’ professional development focusing on thus should be exploited to their fullest. improved classroom effectiveness, and feedback 36 Education Quality and Opportunities for Skills Development in Albania References Engle, P., L. Fernald, H. Alderman, J. Behrman, C. O’Gara, A. Yousafzai, M. Cabral de Mello, M. Hidrobo, N. Ulkuer, I. Ertem, S. Iltus, and Global Child Development Steering Group. 2011. “Strategies for reducing inequalities and improving developmental outcomes for young children in low-income and middle-income countries.” The Lancet- 8 October Vol. 378, Issue 9799: 1339-1353 ). DOI: 10.1016/S0140-6736(11)60889-1. Ferreira, H.G., and J. Gignoux. 2011. “The Measurement of Educational Inequality: Achievement and Opportunity.” IZA Discussion Paper No. 6161. Hanushek, E. 2010. “The High Cost of Low Educational Performance. The long-run economic impact of improving PISA outcomes.” OECD Publications. Hanushek, E., and L. Woessmann. 2006. “Does Educational Tracking Affect Performance and Inequality? Differences-in-differences evidence across countries.” The Economic Journal Vol. 116, Issue 510: C63-C76. Hanushek, E., and L. Woessmann. 2007. “The Role of Education Quality in Economic Growth.” World Bank Policy Research Working Paper 4122. World Bank, Washington, DC. Hanushek, E., and L. Woessmann. 2010. “The Economics of International Differences in Educational Achievement”. NBER Working Paper 15949. Hanushek, E., and L. Woessmann. 2012. “Do Better Schools lead to more growth? Cognitive skills, economic outcomes, and causation.” Journal of Economic Growth Vol. 17: 267-321. Heckman, J. 2008. “Schools, skills, and synapses.” Economic Inquiry 46(3): 289-324. An analysis of PISA 2000-2012 results 37 Heckman, J., and P. LaFontaine. 2010. “The American High School Graduation Rate: Trends and Levels.” Review of Economics and Statistics 92(2): 244–262. OECD. 2012a. “PISA 2009 Technical Report.” Paris: OECD. Retrieved April 10, 2014 from http://www.oecd. org/pisa/pisaproducts/50036771.pdf OECD. 2012b. “PISA 2009 Results: Learning to Learn – Student Engagement, Strategies and Practices (Volume 3).” Paris: OECD. OECD. 2014. “PISA 2012 Results: What Students Know and Can Do.” Paris: OECD. Retrieved April 10, 2014 from http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-I.pdf OECD/United Nations. 2013. “World Migration in Figures.” Retrieved May 18, 2014 from http://www.oecd. org/els/mig/World-Migration-in-Figures.pdf. Sala-i-Martin, X., G. Doppelhofer, and R.I. Miller. 2004. “Determinants of long-term growth: A Bayesian averaging of classical estimates (BACE) approach.” American Economic Review 94 (4), 813-835. World Bank. 2013. “Promoting Excellence in Turkey’s Schools.” World Bank, Washington, DC. World Bank. Forthcoming. “Albania Public Finance Review, Part II: Improving the Efficiency and Quality of Public Spending.” World Bank, Washington, DC. 38 Education Quality and Opportunities for Skills Development in Albania Annex: Empirical Strategy and Results of RIF Regressions The analytical approach used in Section 2 of this report is based on the Firpo, Fortin, and Lemieux (2009) methodology. Typically, the literature on decomposition of student scores in PISA through groups (Amermueller 2004) and years (Barrera et al. 2011) has focused on the mean differences, with little attention to what happens at the tails of the distribution. The Firpo, Fortin, and Lemieux (FFL) method allows one to decompose gaps in student performance not only for the mean but also for other statistics of the distribution. Traditionally, the problem with quantile regressions has been that the law of iterated expectations does not apply, thus making it impossible to interpret the unconditional marginal effect of each independent variable on a student’s performance. However, recent econometric techniques, such as the one proposed by FFL, have solved this methodological difficulty. The FFL technique is based on the construction of re-centered influence functions (RIF) of a quantile of interest as a dependent variable in a regression: ⌧ − D(I  q⌧ )) RIF (I ; q⌧ ) = q⌧ + f I ( q⌧ ) where D is an indicator function and is the density of the marginal distribution of scores. A crucial characteristic of this technique is that it provides a simple way of interpreting the marginal impact of an additional unit of a certain factor on students’ PISA scores. Once the unconditional quantile regression has been computed for different quantiles of the distribution, the results can be decomposed following the Oaxaca-Blinder approach. An analysis of PISA 2000-2012 results 39 Table A.1. Access to school resources by socioeconomic group Bottom ESCS 20% Top ESCS 20% Total Socioeconomic status (Index) -2.38 0.56 -0.95 Teaching practices (Index) -0.61 -0.39 -0.53 Learning strategies (Index) -0.26 0.16 -0.08 Quality of educational resources (Index) -1.11 -0.47 -0.74 Attended one year of preschool (%) 21.6 20.0 22.7 Attended more than one year of preschool (%) 41.6 64.3 52.7 Shortage of qualified teachers (%) 27.8 15.5 19.1 Shortage of computers (%) 58.4 37.7 44.6 Shortage of internet (%) 83.0 47.6 63.3 Shortage of instructional materials (%) 19.0 16.7 17.6 Reading PISA score 346 439 385 Math PISA score 343 421 377 Science PISA score 357 436 391 Source: PISA 2009. Note: Index of teaching practices and learning strategies are normalized at the OECD level. Index of socioeconomic has a full-sample average of -0.55 and a standard deviation of 1.21. The index of quality of educational resources has a full-sample average of -0.28 and a standard deviation of 1.21. Table A.2. Access to school resources by urban/rural residence Urban Rural Total Socioeconomic status (Index) -0.59 -1.40 -0.95 Teaching practices (Index) -0.45 -0.63 -0.53 Learning strategies (Index) 0.00 -0.19 -0.08 Quality of educational resources (Index) -0.45 -1.11 -0.74 Attended one year of preschool (%) 23.1 22.2 22.7 Attended more than one year of preschool (%) 56.1 48.4 52.7 Shortage of qualified teachers (%) 14.5 24.7 19.1 Shortage of computers (%) 35.8 55.2 44.6 Shortage of internet (%) 41.8 89.2 63.3 Shortage of instructional materials (%) 21.2 13.3 17.6 Reading score 409 355 385 Math score 397 354 377 Science score 408 369 391 Source: PISA 2009. Note: Index of teaching practices and learning strategies are normalized at the OECD level. Index of socioeconomic has a full-sample average of -0.55 and a standard deviation of 1.21. The index of quality of educational resources has a full-sample average of -0.28 and a standard deviation of 1.21. 40 Education Quality and Opportunities for Skills Development in Albania Table A.3. Indices of learning strategies and teaching practices Control How students set clear goals for themselves and monitor their own progress in reaching them Memorization Extent to which students try to memorize texts Elaboration How students relate acquired knowledge to other contexts (own life, outside school, and prior knowledge) Learning Strategies Metacognition: Compares students’ strategies for understanding and understanding and remembering with what remembering experts rate as the most appropriate strategies Metacognition: Compares students’ strategies for summarizing summarizing with what experts rate as the most appropriate strategies Discipline, order, and The disciplinary climate in the classroom (noise time management and time taken for students to quiet down) Discussion and debate Extent to which teachers engage students in discussion Relating knowledge Whether teachers help students relate Teaching Practices knowledge to different contexts (prior knowledge and personal experiences) Clarifying Whether teachers outline how student-teacher expectations interaction will be from the beginning Managing Whether teachers mark assignments, check if assignments students understood the lesson, and motivate students Shortage or Science laboratory equipment, instructional Quality of inadequacy of the materials (including textbooks), computers Educational following factors (as for instruction, internet connectivity, computer Resources reported by school software for instruction, library materials, and principals) audio-visual resources Source: See OECD (2012b) for more details on the indices. Note: Indices were constructed by authors by using the coefficients found in a linear regression of reading scores which included the pooled data for all countries participating in PISA 2009. An analysis of PISA 2000-2012 results 41 Table A 4. OLS regressions of PISA 2009 reading scores Variables Albania Full sample Female 42.79*** 19.36*** (3.433) (0.62) Age -14.31** -4.492*** (6.266) (0.972) Grade 0.705 22.05*** (6.203) (0.831) Entrance -8.545** 0.611 (3.849) (0.569) ESCS 13.06*** 8.801*** (1.577) (0.324) ESCSS 28.67*** 31.06*** (6.32) (1.041) Index Teacher Practices 10.39*** 11.71*** (1.9) (0.306) Index Learning Strategies 28.31*** 29.28*** (1.894) (0.379) Student-teacher Ratio 0.858 0.0229 (0.535) (0.0584) Index of Quality of Education Resources 6.91 0.735 (4.242) (0.619) Attended more than one year of pre-primary education 6.071 4.623*** (3.795) (0.726) Proportion of Teachers with Higher Education 27.92** 6.013* (12.8) (3.29) Constant 676.1*** 548.2*** (94.08) (14.91) Country Dummies No Yes Observations 2,812 355,121 R-squared 0.429 0.582 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 Note: Clustered errors at the school level. 42 Education Quality and Opportunities for Skills Development in Albania Table A.5. Oaxaca-Blinder decomposition of performance gap relative to cities, PISA 2009 Variables Town Small Town Village City 438.8*** 438.8*** 438.8*** (7.672) (7.956) (7.882) Location of Interest 364.0*** 386.8*** 411.9*** (7.810) (6.245) (6.999) Difference 74.72*** 51.98*** 26.84** (10.95) (10.11) (10.54) Systemic Efficiency 6.036 14.62 6.400 (11.16) (10.42) (6.857) Explained 68.68*** 37.36*** 20.44** (12.63) (11.62) (9.366) Girl -2.114 -3.288** -1.601 (1.331) (1.380) (1.575) Entrance -0.540 -1.102 -1.642 (0.839) (0.977) (1.065) Age 1.252 1.513 1.248 (0.904) (1.006) (0.896) Grade -0.0250 -2.947 -0.880 (0.321) (2.290) (2.161) ESCS 15.31*** 8.501*** 4.665*** (2.944) (1.950) (1.584) ESCS (School) 42.39*** 25.48*** 15.70*** (10.28) (7.471) (5.335) Teaching Practice (Index) 2.112** 1.964*** 1.719** (0.902) (0.751) (0.781) Learning Strategy (Index) 7.355*** 5.914** 1.949 (2.666) (2.623) (2.394) Quality of School Resources (Index) 4.270 1.938 0.0626 (4.874) (3.083) (1.607) Quality of School Resources^2 (Index) -1.329 -0.614 -0.778 (3.386) (1.098) (2.010) Constant -146.3 90.58 402.2* (254.5) (234.2) (224.0) Observations 1,655 1,597 1,878 Note: Robust standard errors in parentheses. Twofold Oaxaca decomposition. Systemic efficiency accounts for the aggregation of the returns to endowments and unexplained part of the Oaxaca decomposition. Villages are settlements with fewer than 3,000 people; small towns have between 3,000 and 15,000 people; towns have between 15,000 and 100,000 people; and cities have between 100,000 and 1,000,000 people. An analysis of PISA 2000-2012 results 43 Table A.6. Oaxaca-Blinder decomposition of gender reading gap, PISA 2009 Variables Overall Endowments Coefficients Girls 431.1*** (4.549) Boys 379.0*** (5.758) Difference 52.05*** (4.586) Endowments 10.35*** (3.329) Coefficients 42.57*** (3.519) Unexplained -0.859 (1.816) Entrance 1.545* 50.28 (0.801) (46.62) Age 0.0209 -379.0* (0.0896) (228.5) Grade -0.373 6.549* (0.777) (3.638) ESCS -0.599 -5.106 (0.582) (3.133) ESCS (School) -0.191 6.491 (1.250) (7.223) Teaching Practices 1.462** 1.958 (0.674) (1.931) Learning Strategies 8.551*** 0.969 (1.528) (0.664) Student-Teacher Ratio 0.0311 23.63** (0.213) (11.27) Quality of Educational Resources (Index) -0.844 2.565 (0.830) (3.650) Attended at least two years of pre-primary school 0.746 -0.852 (0.601) (2.755) Constant 335.0 (216.7) Observations 2,903 2,903 2,903 Note: Robust standard errors in parentheses. Threefold Oaxaca decomposition. 44 Education Quality and Opportunities for Skills Development in Albania Endnotes 1 Programme for International ability of students to use as that of a student in the 45th Student Assessment. written information in real- percentile in the distribution of 2 See Sala-i–Martin, life situations, including OECD students. Doppelhofer, and Miller (2004). “understanding, using, 10 This pattern is similar when 3 See Hanushek and Woessmann reflecting on and engaging looking at data from PISA 2009. (2007) and Hanushek (2010). with written texts, in order to An 80th percentile student in Using these tests as measures achieve one’s goals, to develop reading in Albania would be a of cognitive skills of the one’s knowledge and potential, 40th percentile student in the population, they show that and to participate in society.” OECD. In math, she would be in countries that had better quality For mathematics literacy, it is the 34th percentile. of education in the 1960s the extent to which students 11 Ferreira and Gignoux experienced faster economic can reproduce mathematical (2011) propose a measure growth during the years 1960- content knowledge, but also of educational opportunity 2000, controlling for other how well they can extrapolate using the share of variance in factors. from what they know and apply test scores that is explained 4 OECD (2012a). their knowledge of mathematics by individual predetermined 5 See Hanushek and Woessmann in both new and unfamiliar circumstances. If a significant (2010) for a detailed review of situations. In both subjects, share of the results is explained analysis of international tests those who are below level 2 can by these characteristics, then like PISA and TIMSS. only answer questions involving the equality of opportunities is 6 Data on enrollment rates come familiar contexts where all low. from UNESCO Institute for relevant information is available 12 See Table A.1 in the Annex. Statistics and were compiled by and the questions are very 13 The World Bank’s mission has EdStats. clearly defined. Such students recently been articulated into 7 One expects children out of can only solve problems that are two main goals: boosting the school to be part of the most almost always obvious and can end of extreme poverty and disadvantaged segments of be immediately extrapolated promoting shared prosperity. society, which tend to have from the given information. The latter is interpreted in this worse performance. 9 An Albanian student in the 80th report to include the bottom 8 PISA defines functional percentile in reading scores 485 40 percent to also capture the literacy in reading as the points, which is the same score people who are considered An analysis of PISA 2000-2012 results 45 moderately poor in middle- 23 The last two processes are income countries such as summarized into a single skill in Albania. PISA student data. 14 See the Annex for the full model. 24 “Average math performance” 15 See Table A.2 in the Annex. refers to the average score 16 After controlling for all of the across all math subscale scores. previous variables, the type of 25 “Average reading performance” school (public or private) does refers to the average score across not significantly affect learning all reading subscale scores. outcomes. 26 Responses on teaching 17 An emigration rate of 29 percent practices and routines and means that for every 100 people student learning strategies for born in Albania, 29 of them live reading are reported by the in OECD countries. students in the PISA Student 18 See OECD/United Nations Questionnaire. (2013). 27 The index is constructed from 19 Individual characteristics and the coefficients of a linear peer effects are usually strongly regression of reading scores correlated and it is not always against the indices in PISA straightforward to disentangle 2009 of practices and strategies the two. separately. 20 See World Bank (2013). 28 See Annex Table A.3 for a 21 See Hanushek and Woessmann description of the variables (2006). and Table A.4 for a summary of 22 See World Bank (forthcoming). regression results. Official statistics and analysis of 29 See Annex Table A.5. the LSMS 2012 offer conflicting 30 See Annex Table A.6. enrollment rate figures: the former is 73 percent and the latter is 33 percent. Education Global Practice Europe and Central Asia Region