Systems Approach for Better Education Results The Learning Crisis in Afghanistan RESULTS OF THE AFGHANISTAN SABER SERVICE DELIVERY SURVEY, 2017 2 | The Learning Crisis in Afghanistan Systems Approach for Better Education Results The Learning Crisis in Afghanistan RESULTS OF THE AFGHANISTAN SABER SERVICE DELIVERY SURVEY, 2017 Authors: Ezequiel Molina, Iva Trako, Anahita Hosseini Matin, Eema Masood and Mariana Viollaz September 2018 Results of the Afghanistan SABER Service Delivery Survey | i Table of Contents Table of Contents ....................................................................................................................... ii List of Acronyms ....................................................................................................................... vii Acknowledgements ..................................................................................................................viii Executive Summary .................................................................................................................... 1 Introduction .............................................................................................................................. 15 Framework — World Development Report 2018 .................................................................... 17 Background of SABER Service Delivery Survey......................................................................... 19 Chapter 1 : Is Afghanistan’s Education System Aligned with Learning? .................................. 21 Section II. Student Results: Mathematics Assessment .................................................................................... 30 Section III. Curriculum-adjusted Years of Schooling: Methodology and Findings Using Actual Questions ..... 34 Section IV. Disaggregation by Urban vs. Rural Schools .................................................................................... 40 Section V. Disaggregation by Gender-Mix: Boys, Girls, Co-ed Schools, Co-ed Classroom Schools) ................. 43 Section VI. Looking Beyond the Mean ............................................................................................................. 45 Chapter 2 : Teachers ................................................................................................................. 57 Section I. What Do teachers Do? Understanding Teachers Effective Use of Time ......................................... 60 Section II. What Do Teachers Know? Measuring Teachers’ Content Knowledge in Language and Mathematics ......................................................................................................................................... 63 Section III. How Well Do Teachers Teach? Teachers’ Pedagogic Content Knowledge and Teaching Skills ..... 72 Section IV. What Explains These Results? Why Does the System Used to Select, Train, and Remunerate Teachers not Produce High Quality Teaching? ..................................................................................... 82 Chapter 3 : School Management .............................................................................................. 89 Section I. What Do Principals Do? Understanding Principals’ Professional Development and Effective Use of Time ...................................................................................................................................................... 91 Section II. What Do Principals Know? Principals’ Ability in School Management ........................................... 98 Section III. Principals’ Decision-Making Power .............................................................................................. 105 Section IV. Accountability: District Supervision Visits and Community Participation .................................... 113 Chapter 4 : Schools Inputs ...................................................................................................... 117 Section I. Inputs for Learning — School Level ................................................................................................ 118 Section II. Inputs for Learning — Classroom Level ......................................................................................... 125 Section III. School Inputs — International Comparison ................................................................................. 129 Chapter 5 : Student Support................................................................................................... 131 Chapter 6 : Community-Based Education in Afghanistan ..................................................... 149 References .............................................................................................................................. 171 ii | The Learning Crisis in Afghanistan Appendix ................................................................................................................................. 174 Appendix A.1: Afghanistan SABER SD Sampling Methodology ...................................................................... 174 Appendix A.2: The Afghanistan SABER SD Test Instrument ........................................................................... 177 Appendix B.1: Psychometric Properties ......................................................................................................... 180 Appendix B.2. Robustness Checks for Curriculum-adjusted years of schooling Definitions ......................... 192 Appendix C: Additional Tables ....................................................................................................................... 200 List of Tables Table 1.1: Language questions and corresponding grade level in Afghanistan .................................................................. 24 Table 1.2: Mathematics questions and corresponding grade level in Afghanistan ............................................................ 32 Table 1.3: Number of schools by type of school and urban/rural ...................................................................................... 43 Table 1.4: Student learning performance in language by type of school – Grade 4 ........................................................... 44 Table 1.5: Student learning performance in mathematics by type of school – Grade 4 .................................................... 45 Table 1.6: R-Squared- Variance decomposition of student’s learning scores in Mathematics and Language .................. 48 Table 1.7: Descriptive statistics of schools — Afghanistan 2017 ........................................................................................ 49 Table 1.8: Average school type and school facilities – Afghanistan 2017 .......................................................................... 50 Table 1.9: Comparing Afghan students internationally in Mathematics – Grade 4............................................................ 53 Table 2.1: Comparison of teachers’ time use ..................................................................................................................... 61 Table 2.2: How does the average Afghan teacher looks like? ............................................................................................ 62 Table 2.3: Comparison of teachers’ performance on Language ......................................................................................... 65 Table 2.4: Comparison of teachers’ performance on Mathematics ................................................................................... 71 Table 2.5: Comparison of teachers’ performance on pedagogic knowledge and skills in the classroom .......................... 75 Table 2.6: Strengths and weakness of the teachers’ support system ................................................................................ 86 Table 3.1: Principals’ use of time per day by urban/rural .................................................................................................. 93 Table 3.2: Number of primary school leadership positions by urban/rural ....................................................................... 94 Table 3.3: Type of supervision visits as reported by the principal.................................................................................... 113 Table 4.1: Number of classrooms per school by urban/rural schools .............................................................................. 120 Table 4.2: School input availability by gender-mix at the school level (boys/girls/coed schools) .................................... 121 Table 4.3: Classroom and teaching input availability by school type (boys/girls/coed schools) ...................................... 127 Table 4.4: School and classroom inputs availability — international comparison ........................................................... 130 Table 5.1: Comparing availability of learning equipment internationally ........................................................................ 134 Table 5.2: Background characteristics of Afghan students............................................................................................... 140 Table 5.3: Absence rate and student dedication to learning............................................................................................ 147 Table 5.4: Student Effort Indicators Internationally ......................................................................................................... 147 Table 6.1: Teachers, school inputs and students: Public schools versus CBE Schools ...................................................... 166 Table 6.2: Comparison of teachers’ performance on pedagogic knowledge and skills in the classroom......................... 168 Table 6.3: Strengths and weakness of the teachers’ support system .............................................................................. 169 Table A.1.1: Tradeoffs in statistical precision ................................................................................................................... 176 Table A.2.1: Afghan curriculum example ......................................................................................................................... 177 Table B.1.1: Internal consistency of the student and teacher assessment ...................................................................... 180 Table B.1.2: Item characteristic curves for the Mathematics Assessment ....................................................................... 183 Table B.1.3: Item characteristic curves for the Mathematics Assessment ....................................................................... 185 Table B.1.4: Item characteristic curves for the Language Assessment ............................................................................. 188 Table B.1.5: Item characteristic curves for the Mathematics Assessment ....................................................................... 190 Table B.1.6: Item characteristic curves for the Pedagogy Assessment ............................................................................ 191 Table B.2.1: Number of questions skipped and number of students who skipped them ................................................ 194 Table C.1: Student’s learning performance in language by type of language (Dari vs. Pashtu) – Grade 4 ....................... 200 Results of the Afghanistan SABER Service Delivery Survey | iii Table C.2: Student’s learning performance in language by urban/rural schools – Grade 4 ............................................. 201 Table C.3: Student’s learning performance in mathematics by urban/rural schools – Grade 4 ....................................... 202 Table C.4: Language questions and percentage of skipped questions by gender – Grade 4 ............................................ 203 Table C.5: Mathematics questions and percentage of skipped questions by gender – Grade 4 ...................................... 204 Table C.6: Comparison of teachers’ time use ................................................................................................................... 205 Table C.7: Teachers’ content knowledge in language by type of task .............................................................................. 205 Table C.8: Teachers’ performance in mathematics by type of task.................................................................................. 206 Table C.9: Pedagogical knowledge and skills in the classroom......................................................................................... 206 Table C.10: Strengths and weakness of the teachers’ support system ............................................................................ 208 Table C.11: School input availability by urban/rural schools ............................................................................................ 209 Table C.12: Classroom and teaching input availability by urban/rural distribution (%) ................................................... 210 Table C.13: Comparison of student support provided by school ..................................................................................... 211 Table C.14: Comparison of student support available from home................................................................................... 212 Table C.15: Comparison of dedication to learn ................................................................................................................ 213 Table C.16: Student’s learning performance in language by public school/CBE schools – Grade 4 ................................. 213 Table C.17: Student’s learning performance in mathematics by public school/ CBE schools – Grade 4 .......................... 214 List of Figures Figure 0.1: Distribution of curriculum-adjusted years of schooling in Afghanistan...................................................................................... 2 Figure 0.2: Afghan student’s performance by type of school ...................................................................................................................... 3 Figure 0.3: Curriculum-adjusted years of schooling by public schools/CBE Schools .................................................................................. 11 Figure 0.4: Why learning doesn’t happen (WDR 2018) .............................................................................................................................. 17 Figure 1.1: Samples of Language questions ............................................................................................................................................... 25 Figure 1.2: Student learning performance in Language – Grade 4 ............................................................................................................. 27 Figure 1.3: Student learning performance in language by type of language (Dari vs Pashtu) – Grade 4 ........................................................ 29 Figure 1.4: Samples of Mathematics questions ......................................................................................................................................... 30 Figure 1.5: Student learning performance in Mathematics – Grade 4 ....................................................................................................... 33 Figure 1.6: Distribution of curriculum-adjusted years of schooling – Language ........................................................................................ 34 Figure 1.7: Distribution of curriculum-adjusted years of schooling - Mathematics ................................................................................... 36 Figure 1.8: Learning progression in Mathematics concepts ....................................................................................................................... 39 Figure 1.9: Student’s learning performance in language by urban/rural schools ...................................................................................... 41 Figure 1.10: Student’s learning performance in mathematics by urban/rural schools .............................................................................. 42 Figure 1.11: Differences in learning outcomes between and within schools — Mathematics................................................................... 46 Figure 1.12: Differences in learning outcomes between and within schools — Language ........................................................................ 47 Figure 1.13: Distribution of student-teacher ratios - Afghanistan 2017..................................................................................................... 50 Figure 1.14: Distribution of observed student-teacher ratios - Afghanistan 2017 ..................................................................................... 51 Figure 1.15 : Comparing Afghanistan students internationally in Mathematics – Grade 4 ........................................................................ 54 Figure 1.16 : Comparing Afghanistan students internationally in Language – Grade 4 .............................................................................. 55 Figure 2.1: What do teachers do? .............................................................................................................................................................. 61 Figure 2.2: Teachers’ content knowledge in Language by type of task ...................................................................................................... 64 Figure 2.3: TIMSS Item used as part of the Teacher Language Assessment ............................................................................................... 65 Figure 2.4: TIMSS Item used as part of the Teacher Language Assessment ............................................................................................... 66 Figure 2.5: Teachers’ performance in Mathematics by type of task .......................................................................................................... 67 Figure 2.6: Teachers’ content knowledge item — Peter’s Telephone Bill .................................................................................................. 68 Figure 2.7: Distribution of curriculum-adjusted years of schooling – Mathematics teachers .................................................................... 70 Figure 2.8: Peter’s (Parvez) Telephone Bill Exercise ................................................................................................................................... 72 Figure 2.9: Pedagogical knowledge and skills in the classroom ................................................................................................................. 74 Figure 2.10: Distribution of CLASS domains scores .................................................................................................................................... 78 iv | The Learning Crisis in Afghanistan Figure 2.11: Distribution of CLASS dimension scores ................................................................................................................................. 81 Figure 2.12: Mismatch between what teachers study and what they teach — Language ......................................................................... 82 Figure 2.13: Mismatch between what teachers study and what they teach – Mathematics ..................................................................... 83 Figure 2.14: Classroom practices by training level ..................................................................................................................................... 84 Figure 3.1: Primary school principals receiving management training and average duration ................................................................... 91 Figure 3.2: Principal’s professional development ...................................................................................................................................... 92 Figure 3.3: Hours per day principals spend in school and use of time ....................................................................................................... 93 Figure 3.4: Principal and head teacher absence rate ................................................................................................................................. 94 Figure 3.5: Percentage of male and female principals/deputy principals/head teachers .......................................................................... 95 Figure 3.6: Principals’ level of education in Afghanistan............................................................................................................................ 95 Figure 3.7: Positions before becoming primary school principal ............................................................................................................... 96 Figure 3.8: Distribution of principal’s age in Afghanistan .......................................................................................................................... 96 Figure 3.9: Distribution of Years of Experience before becoming a Principal by Gender and Rural/Urban .............................................. 97 Figure 3.10: Comparison of Principal’s and Teacher’s Monthly Salary in Afghanistan ............................................................................... 98 Figure 3.11: Principal knowledge of school performance in Language and Mathematics ....................................................................... 100 Figure 3.12: Principal knowledge about students performance in single digit addition .......................................................................... 101 Figure 3.13: Principal knowledge about teacher knowledge in double digit subtraction ........................................................................ 102 Figure 3.14: Principals practices of teacher evaluation............................................................................................................................ 103 Figure 3.15: Principals’ opinion on teachers’ most important responsibility ........................................................................................... 105 Figure 3.16: Principal decision-making power ......................................................................................................................................... 106 Figure 3.17: Principal’s main reasons for sending a teacher for training ................................................................................................. 108 Figure 3.18: Distribution of decision making power in school management areas.................................................................................. 109 Figure 3.19: Actual decision-making cases for the afghan principals ....................................................................................................... 110 Figure 3.20: Principal’s main constraints to improve student’s learning ................................................................................................. 112 Figure 3-21: Frequency of supervision visits per school year ................................................................................................................... 113 Figure 3.22: Community involvement ...................................................................................................................................................... 114 Figure 3.23: Main areas of discussion with the SMC and PTA.................................................................................................................. 115 Figure 4.1: Afghanistan school infrastructure by urban/rural schools ..................................................................................................... 119 Figure 4.2: Main sources of drinking water in Afghan schools ................................................................................................................. 123 Figure 4.3: Type of roads to access schools in Afghanistan ..................................................................................................................... 124 Figure 4.4: Classroom and teaching input availability by urban/rural schools ......................................................................................... 126 Figure 5.1: Minimum school equipment availability by urban/rural school (%)....................................................................................... 133 Figure 5.2: Minimum school equipment availability by type of school (%) .............................................................................................. 134 Figure 5.3: Students/Classrooms with special needs and PTSD support by urban/rural school .............................................................. 135 Figure 5.4: Students/Classrooms with special needs and PTSD support by type of school ...................................................................... 136 Figure 5.5: Supportive behavior in the classroom by urban/rural schools ............................................................................................... 137 Figure 5.6: Supportive behavior in the classroom by school type in terms of gender mix ....................................................................... 138 Figure 5.7: Supportive behavior in the classroom by teacher gender ...................................................................................................... 138 Figure 5.8: Supportive behavior in co-education classrooms by student gender .................................................................................... 139 Figure 5.9: Who Do the students live with? — Afghanistan .................................................................................................................... 141 Figure 5.10: Parental education in Afghanistan ....................................................................................................................................... 142 Figure 5.11: Household assets by urban/rural schools ............................................................................................................................ 143 Figure 5.12: Distance from school ........................................................................................................................................................... 144 Figure 5.13: Home support by urban/rural schools ................................................................................................................................. 144 Figure 5.14: Student engagement using CLASS ........................................................................................................................................ 146 Figure 6.1: Provinces covered in the Afghanistan SABER SD by public/CBE schools ................................................................................ 152 Figure 6.2: Student’s learning performance in Language by public/CBE school – Grade 4 ...................................................................... 154 Figure 6.3: Student’s learning performance in Language by public/CBE school and gender – Grade 4 ........................................................ 155 Figure 6.4: Student’s learning performance in Mathematics by public/CBE schools – Grade 4 ............................................................... 157 Figure 6.5: Student learning performance in Mathematics by public/CBE schools and gender – Grade 4 ............................................. 158 Figure 6.6: Curriculum-adjusted years of schooling by public/CBE schools – Grade 4 ............................................................................. 159 Figure 6.7: Distribution of curriculum-adjusted years of schooling in Language by public/CBE schools – Grade 4 ................................. 160 Results of the Afghanistan SABER Service Delivery Survey | v Figure 6.8: Distribution of curriculum-adjusted years of schooling in Mathematics by public/CBE schools – Grade 4 ........................... 161 Figure 6.9: Comparing Afghanistan’s CBE schools internationally ........................................................................................................... 162 Figure 6.10: The learning gap in Language by gender, school type and socioeconomic backgrounds ......................................................... 163 Figure 6.11 : The learning gap in Mathematics by gender, school type and socioeconomic backgrounds ............................................. 164 Figure 6.12: Teacher content knowledge: Public schools versus CBE schools ......................................................................................... 167 Figure B.1.1: Item characteristic curve description of an item from the test........................................................................................... 181 Figure B.1.2: Item characteristic curve for the Language Assessment ..................................................................................................... 182 Figure B.1.3: Item characteristic curve for the Mathematics Assessment ............................................................................................... 184 Figure B.1.4: Item characteristic curve for the Language Assessment (First 22) ...................................................................................... 186 Figure B.1.5: Item characteristic curves for the Language Assessment (22-44) ....................................................................................... 187 Figure B.1.6: Item characteristic curves for the Mathematics Assessment.............................................................................................. 189 Figure B.1.7: Item characteristic curves for the Pedagogy Assessment ................................................................................................... 191 Figure B.2.1: Robustness check— distribution of curriculum- adjusted years of schooling in Language – Grade 4 ................................. 192 Figure B.2.2: Robustness Check— Distribution of curriculum- adjusted years of schooling in Mathematics – Grade 4 .......................... 193 Figure B.2.3: Questions more likely to be skipped in Language – Grade 4 ............................................................................................... 195 Figure B.2.4: Questions more likely to be skipped in Mathematics – Grade 4 ......................................................................................... 196 Figure B.2.5: Curriculum-adjusted years of schooling in Language — original versus excluding missing values – Grade 4 .................... 197 Figure B.2.6: Curriculum-adjusted years of schooling in Mathematics — original versus excluding missing values – Grade 4 ............... 197 Figure B.2.7: Observed and predicted distribution of curriculum-adjusted years of schooling in Language – Grade 4 .......................... 198 Figure B.2.8: Observed and predicted distribution of curriculum-adjusted years of schooling in Mathematics – Grade 4 .................... 199 List of Boxes Box 0.1: Applying the WDR Framework to CBE Schools ..................................................................................................... 10 Box 0.2: Priority Areas for Action ....................................................................................................................................... 14 Box 0.3: The Link between SABER SD and SDI Approach ................................................................................................... 20 Box 1.1: What does a typical school in Afghanistan look like? ........................................................................................... 49 Box 1.2: The Learning Crisis: Comparing Afghanistan Students Internationally ................................................................. 53 Box 2.1: Profile of Afghanistan’s Teachers ......................................................................................................................... 62 Box 2.2: A Tale of Three Classes ......................................................................................................................................... 76 Box 2.3: Pre-Service Teacher Training is not associated with better teaching skills .......................................................... 84 Box 2.4: Notes from the Field ............................................................................................................................................. 87 Box 3.1: Profile of Afghanistan’s Principals ........................................................................................................................ 95 Box 3.2: Principal’s letters to The World Bank ................................................................................................................. 111 Box 4.1: Notes from the field on school infrastructure in Afghanistan ............................................................................ 124 Box 5.1: Comparing Availability of Learning Equipment Internationally .......................................................................... 134 Box 5.2: What does a typical student in Afghanistan look like? ....................................................................................... 140 Box 5.3: Notes from the Field ........................................................................................................................................... 145 Box 5.4: Comparing Student Effort Indicators Internationally ......................................................................................... 147 Box 6.1: Sampling Design for CBE schools ........................................................................................................................ 152 vi | The Learning Crisis in Afghanistan List of Acronyms AFG Afghanistan AFN Afghan Afghani currency CBE Community-Based Education CLASS Classroom Assessment Scoring System EQUIP Education Quality Improvement Project GDP Gross Domestic Product GER Gross Enrollment Rate ISKP Islamic State in Khorasan Province LEAPS Learning and Educational Achievements in Punjab Schools MoE Ministry of Education NGO Non-Government Organization OECD Organization for Economic Co-operation and Development PED Provincial Education Department PDR People’s Democratic Republic PIRLS The Progress in International Reading Literacy Study PISA Programme for International Student Assessment PTA Parents Teachers Association PTSD Post-Traumatic Stress Disorder SABER Systems Approach for Better Education Results SABER SD SABER Service Delivery SMC School Management Committee SSA Sub-Saharan Africa TIMSS Trends in International Mathematics and Science Study TVET Technical and Vocational Education and Training UNESCO The United Nations Educational, Scientific and Cultural Organization USD United States Dollars WDR World Development Report Results of the Afghanistan SABER Service Delivery Survey | vii Acknowledgements This report was prepared by a team led by Ezequiel Molina. The core team included Iva Trako, Anahita Hosseini Matin, Eema Masood and Mariana Viollaz. Additionally, Mario Vaisman, Owen Ozier, Rohit Chhabra, Tracy Wilichowski, and Halsey Rogers were part of the extended team. The team is especially thankful to the Afghanistan World Bank Education Task Team Leader Nathalie Lahire for initiating the study and leveraging the support from government and development partners, and Matiullah Noori, Ahmad Khalid Afridi, Ahmad Rafi Otofat from the Country Office Team for the support on the ground. The team is grateful to Restituto Cardenas, Jr. for his help compiling the report. Data collection was undertaken by Rahman Safi International (RSI) consulting firm consisting of Margaret Orwig and Inayat Rahman (Managing Partners), Sadam Hussain (Project Manager), Sayed Wakil (Data Base Manager), Sohail Ahmad, Nazif Delir, Navid Zia, Jawad Ahmad, Nasibullah Nasib, and Ahmad Qais (Regional Team Supervisors), Jamil Rahman (Data Base Officer), and Altaf Wahid (Data Quality Assurance Officer). Further data collection was performed by Ezatullah, Khairullah, Mohammad Arif, Sayed Abas, Esmatullah Masomi, Ahmad Shah Patsoon, Sefatullah Sahil, Zabiullah Saifi, Mohammad Tahir Mominyar, Sayed Najiullah Sayedzada, Shoaibullah Abeed, Abdul Qahir, Sayed Muhiburab Sadaat, Himatullah Safi, Nisar Ahmad Safi, Farhad Shafaq, Abdul Satar, Nematullah Esazada, Jamaludin Delawary, Hamza Paiman, Jalil Ahmad Mohammadi, Hamidullah Akbari, Habiburahman Hanif, Javid Rahman Uosafi, Rafi Ullah, Asadullah Omarzai, Mohammad Turyalai Sharifzai, Nasir Ahmad Ahmadi, Khalil U Rahman Bawar, Mohammad Tahir, Aminullah Safi, Naqeebullah Fazali, Ahmad Zaari, Behroz Sadri, Ashuqllah, Sardar Khan Rahmanzai, Parweez Ahmad Nisar, Hashmat Khan, Mehdi Hamdard, Zabeehullah Fazli, Zahurullah, Nik Mahmmad, and Mohammad Hamid (enumerator). The team would also like to thank the officials of the Afghanistan Ministry of Education, in particular Deputy Minister of General Education Dr. Mohammad Ibrahim Shinwari, Susan Wardak (Director of Teacher Education), and Dr. Amir (EQUIP acting director) and the peer reviewers of the study, Marina Bassi, Deon Filmer, and Yasuhiko Matsuda. Finally, the team would like to extend their sincere gratitude to World Bank Afghanistan Country Management Unit, especially Shubham Chaudhuri (Country Director), Yasuhiko Matsuda (Program Leader), and Stephen Ndegwa (Former Operations Manager), for their guidance and support. The team is also grateful to the security team led by Peter Campbell for guaranteeing our team safety during the field visits. viii | The Learning Crisis in Afghanistan Executive Summary Schooling is not learning. Although access to schooling has improved significantly in the last decade, Afghan students are not learning. After spending 4 years in primary school, around 65% of Afghan students have only fully mastered Grade 1 Language curriculum and less than half of them mastered Grade 1 Mathematics curriculum. Data collected from the SABER SD survey show that Afghan students could correctly answer only 30% of the questions on the Language test, on average. In particular, one- third could not identify a picture from a given word, three-quarters could not form a sentence with the verb “went” or the verb “is cooking”, and less than 15% could comprehend a simple paragraph. Their performance in Mathematics is even more worrisome; Afghan students scored an average of only 25% on the Mathematics test. Although most Afghan students could add single and double digit numbers and subtract single digit numbers, they were unable to subtract double digit numbers or complete triple digit equations. Most students lack both multiplication and division skills, and almost none could solve word problems, compute fractions, identify shapes, or calculate an area. In other words, even though a Grade 4 student has been in the system for 4 years, they only display the knowledge of a Grade 1 student. More specifically, just 65% of the Grade 4 students have mastered the Language curriculum for Grade 1 and only 15% could perform grade-4 Language questions. Similarly, in Mathematics, less than half the students have mastered the Grade 1 mathematics curriculum. Moreover, less than 3% of students could solve grade-4 Mathematics questions (Figure 0.1). Results of the Afghanistan SABER Service Delivery Survey | 1 Figure 0.1: Distribution of curriculum-adjusted years of schooling in Afghanistan Note: In Grade 4, 65% of the tested students have fully mastered the Language curriculum and less than half the Mathematics curriculum for Grade 1. Source: SABER Service Delivery - Afghanistan 2017 The most variation in learning is found across schools in the same district/village. However, even within the same school and classroom, Afghan students are at very different levels of learning. This large variation across schools implies that in the same district or village, there are schools where every student can at least perform single and double digit addition or identify a picture from a word, and schools where the vast majority of children cannot. Moreover, Afghan students in the top performing schools present a wide spectrum of learning in the same class, ranging from students who cannot perform the most basic mathematics tasks to those who are close to performing word problems. Afghan same-sex schools outperformed the co-educational schools in both subjects. Co-educational classrooms tended to score lowest. Four types of schools were observed in Afghanistan: boys schools (25%); girls schools (30%); co-ed schools with same-sex classrooms (30%), e.g. boys and girls sit in different classroom within the same school; and co-ed schools with co-ed classrooms (15%) e.g. boys and girls sit in the same classroom. In Language, girls schools tested similarly to boys schools, and performed overwhelmingly better than both types co-educational schools. Boys schools do not differ from girls schools and also from co-educational schools in the school, but they perform better than co- educational schools in the classroom. Lastly, language performance in co-educational schools is indistinguishable between genders. In Mathematics, Afghan students in same-sex school have better results than those in co-educational schools (Figure 0.2). 2 | The Learning Crisis in Afghanistan Figure 0.2: Afghan student’s performance by type of school Boys School (25%) Girls School (30%) Co-ed in School (30%) Co-ed in Class (15%) SAME – SEX SCHOOLS CO-EDUCATIONAL SCHOOLS Afghan students in urban schools tend to outperform those in rural schools in both Mathematics and Language. For most of Grade 4 questions in the Language test, Afghan students in urban schools showed significantly better results than those in rural schools, especially in harder tasks such as reading comprehension. In Mathematics, urban schools usually outperformed rural schools or showed similar results to rural schools. In particular, Afghan students in urban schools perform significantly better on word problems and on other relatively difficult tasks such as double and triple digit subtraction. This result is in line with Figure 0.2 as classroom co-educational schools are more likely to be located in rural areas. Afghan students lag significantly behind the TIMSS and Pre-PIRLS international average in Mathematics and Language. Their performance is comparable to students in Sub-Saharan African countries. In Mathematics, Afghan student performance is comparable to the average student performance across Sub-Saharan Africa--in fact, the results are on par with Tanzania and Togo. Afghan students significantly outperform students in Mozambique but underperform relative to Kenya. Moreover, they consistently performed below the TIMSS International average. In Language, Afghan students underperform relative to Pakistan, Botswana, Colombia, and South Africa and are below the Pre-PIRLS international average. WHY IS AFGHANISTAN’S EDUCATION SYSTEM NOT ALIGNED WITH LEARNING? The WDR 2018 provides a theoretical lens to examine why an education system may not be aligned with learning. In particular, ineffective education systems often lack one or more of the four key school-level determinants of improved learning: skilled and motivated teachers, effective school management, school inputs that affect teaching and learning, and prepared and supported students. Results of the Afghanistan SABER Service Delivery Survey | 3 TEACHERS While teachers in Afghanistan have low absence rates and high time on task, they struggle to help student learn as they have very low content and pedagogical knowledge, as well as poor teaching skills. What do teachers do? 15% of teachers were absent from class, either because they were absent from school entirely or in the school but not in the classroom. While in the classroom, teachers spent 80% of their time on learning activities, which is close to the 85% target. Factoring teacher absence from school and class, as well as the percentage of the lesson devoted to non- teaching activities, students are taught, on average, 2 hours and 18 minutes per day out of a 3 hour and 25-minute school day. While teachers have low absence rates and high time on task, the short scheduled teaching time means Afghan students receive less effective teaching than the average Sub-Saharan African country (the dataset includes Kenya, Mozambique, Togo, Tanzania, Senegal, Uganda, and 4 Nigerian states). What do teachers know? Teachers were asked to mark (or “grade”) mock student tests in language and in Mathematics. The exercise assessed their ability to spell simple words (“traffic,” for example), identify the correct grammatical option that would complete a sentence such as “[ ] [Who, How much, How many] oranges do you have?”. On average, language teachers correctly answered these questions 75% of the time. They also had to correct spelling, grammar, syntax, and punctuation mistakes in a child’s letter that included segments such as “I went to tell you that my new school is better than the old one I have a lot of thing to tell you about my new school in Kabul”. Teachers struggled with this, earning only 30% of available points. The test also included Cloze passages, which included “student” responses such as “[Where] do I have to go to the market?” (In this case, a correct answer could be either “Why or When.”). Teachers scored an average of 44% in this exercise. Finally, they were asked to read a passage from the Lonely Giraffe and answer two reading comprehension questions. The average score of both questions was 44 out of 100, showing teachers lack basic reading comprehension skills we expect grade 4 students to have mastered. In Mathematics, we tested if the teacher can accurately correct children’s work in such aspects of numeracy as manipulating numbers using whole number operations. In essence, these measured whether the math teacher masters his or her students’ curriculum. Fewer than 40% of Mathematics teachers showed evidence of mastering Grade 4 curriculum. Looking at specific tasks in Mathematics, almost 2 in 10 teachers cannot add double digits, a quarter of the teachers cannot do a single division, and more than one-third of the teachers cannot multiply triple digit numbers. 4 | The Learning Crisis in Afghanistan How well do teachers teach? To assess how well teachers teach, we measure: (i) teachers’ pedagogical knowledge, (ii) teachers’ capacity to assess students and monitor their progress, (iii) teachers’ actual use of a monitoring system and knowledge of student performance; and (iv) teachers’ classroom practices based on direct lesson observation. To measure pedagogical knowledge, we asked teachers to prepare a lesson plan by reading and extracting information from a factual text on a topic and to state what they would expect their students to learn from the lesson. While teachers struggle to read and understand the text (average score of 39%), they struggled even further to formulate what they wanted children to learn from the lesson based on their reading (average score of 30%). To measure teachers’ ability to assess students’ learning and give feedback, teachers were asked to: (i) prepare lower and higher order questions, (ii) use a marking scheme to give feedback on strengths and weaknesses in students’ writing, and (iii) use a list of students’ grades to turn the raw scores into averages and comment on the student’s learning progression. Few teachers demonstrated ability in at least one of these areas (average score of 25%), and no teacher in the sample could answer at least 80% of the items in the three tasks correctly. To go beyond capacity and measure whether teachers implement a monitoring system, we asked teachers: (i) to show us their records of individual students’ performance through the year, and (ii) to estimate their students’ knowledge in three areas of their curriculum (e.g. percentage of students that could correctly answer a single digit addition question). We then compare their answer with the scores from the student assessment module. Less than a third of teachers keep a record of students’ performance and even fewer were able to estimate their performance in those selected areas within a twenty-point margin. No teacher in the sample could estimate all three areas correctly. Poor knowledge of pedagogy was mirrored in teacher behavior in the classroom. Overall, few (less than two in ten) deploy the teaching practices identified in the literature as promoting learning—structuring, planning, asking lower and higher order questions and giving feedback—in their lessons. If “adequate” instruction requires students to be taught by teachers with at least minimal pedagogic knowledge and subject knowledge for at least three hours a day, then, essentially, no public primary school in Afghanistan currently offers an adequate education. These results point to two systemic challenges: the system used to select and train teachers does not deliver high-quality candidates; and the system used to monitor and support teachers does not help them deliver high-quality teaching. Results of the Afghanistan SABER Service Delivery Survey | 5 SCHOOL MANAGEMENT Although Afghan principals work long hours and have low absence rates, they are unaware of basic performance indicators, such as percentage of students in Grade 4 that can add double digit numbers or their own teacher content knowledge. This could in part be explained by poor preparation and support they received to lead the school. What do principals do? The majority of primary school principals (83%) reported receiving school management training on administrative skills; however, almost none received training on helping teachers improve instruction. The largest proportion of their time, which is a little less than 8 hours, is used to perform school management activities as managing teachers, managing the school administration, and asset management. In addition to completing their duties as principal, they use 17% of their time to teach. This is equivalent to around 1 hour and a half of teaching per week. In sum, Afghan principals are not equipped to provide any pedagogical support and coaching to teachers. What do principals know? Most Afghan principals (97%) were not knowledgeable of their schools’ performance, in terms of teacher absence, teacher content knowledge, and learning outcomes. In particular, nearly all Afghan principals thought the majority of their grade 4 students could solve a single digit addition problem; only 41% of principals correctly predicted the ability of their students. Similarly, almost all Afghan principals thought most of their teachers could solve a double digit subtraction problem; only 35% of principals correctly predicted the ability of their teachers. Only 13% of Afghan principals follows good practices of teacher evaluation, which include meeting with teachers to evaluate their performance, using information from student assessments and classroom observations to assess performance, and providing regular feedback to teachers. Even with proper teacher evaluation, principals do not have the proper training provide pedagogical support for teachers as principals’ professional development focuses on administrative skills. Almost all principals in Afghanistan reported that teachers’ most important responsibility is being on time to school (88%), followed by maintaining strict discipline in the classroom (7%), and lastly teaching students to be good citizens (5%). Surprisingly, none of the principals considered improving students’ learning to be the primary focus of teachers’ duties. Mismatch between principals’ skills and decision-making power Principals in Afghanistan lack in general autonomy and decision-making power. When they do have some autonomy, they don’t have the skills to make good decisions. For example, Afghan principals report having a lot of decision-making power when appointing a teacher to participate in training, but less than half of the principals have taken the necessary trainings that would allow them to make a good decision about which training teachers should do. 6 | The Learning Crisis in Afghanistan SCHOOL INPUTS Public schools in Afghanistan lack basic necessary infrastructure and do not provide students with an adequate learning environment in the school or in the classroom. Afghanistan has made significant strides in improving primary education enrollment in the past 15 years. In 2001, one million Afghan students, almost none of them girls, were enrolled in 3,400 schools. In 2015, there has been a nine-fold increase in enrollment with more than eight million students in 16,400 schools, of whom almost 40% are girls.1 Although encouraging to see progress in access to education, this expansion has not kept pace with the availability of school inputs in Afghanistan. What school inputs are available in schools? Despite the surge in reconstruction of schools in Afghanistan through programs such as EQUIP 2, only one-third (35%) of public schools have minimum infrastructure availability, which includes the availability of functioning toilets and classroom visibility. Schools in Afghanistan are not equipped with sufficient number of classrooms. Less than three-quarters of the schools in the sample (70%) have permanent classrooms (i.e. complete, with floors and walls), 15% have semi-permanent classrooms (i.e. incomplete, plastered without shutters or floor, etc.), and 15% of the schools have temporary classrooms. The ratio of students per class is close to the norm of 1:40, but in urban schools, classrooms can be very overcrowded. Furthermore, shortcomings terms of availability of toilets, drinking water, and electricity in Afghanistan are pervasive. Overall, almost half (46%) of Afghan schools do not have at least one functioning toilet. Even in schools with such facilities, there are around 169 students per every toilet. This number can be even higher in urban schools and especially girls’ schools, in which the ratio can be as high as 231 students per toilet. In addition, around 70% of public schools lack clean drinking water and working electricity. In Afghanistan, 80% of the schools have boundary walls and 77% have a security guard at the door. Urban schools are better protected compared to rural schools. Despite Afghanistan’s fragility, only 10% of public schools have a safe shelter to protect students. Lastly, on a national scale, 40% of public schools own at least a computer, but only 5% of these computers are connected to internet. Urban schools and boys’ schools are generally better equipped with computers compared to other types of schools. 1 Education Quality Reform in Afghanistan Project Appraisal Document (World Bank, 2017) 2 Education Quality Improvement Program (EQUIP) is a World Bank project that aims to increase equitable access to quality basic education for students in Afghanistan. EQUIP is implemented by the Afghanistan Ministry of Education and funded by the Afghanistan Trust Fund (ARTF). EQUIP supports building new school infrastructures as well as improving school facilities (e.g. building library, extra classrooms, laboratory, computer purchase, etc.) and teaching materials. Results of the Afghanistan SABER Service Delivery Survey | 7 What classroom inputs are available in schools? Classrooms in Afghanistan also lack the necessary inputs to provide students with an appropriate learning environment. Overall, 35% of the students do not have proper seats and desks. On the positive side, around 89% of the classrooms are equipped with a functioning blackboard and almost all schools have chalk and/or marker available in the classrooms. Around 80% of classrooms in Afghanistan are sufficiently visible to the students, meaning that it was possible for students to read a printout placed on the blackboard from the front as well as from the back of the classroom. Corner libraries in the classrooms are practically non-existent in Afghanistan, where only 1% of urban classrooms are equipped with such facilities. On average, one-third of Afghan classrooms displayed some kind of educational material on the walls, such as artworks, charts, maps, etc. In terms of classroom hygiene, 8% of the classrooms were extremely clean and well maintained, 80% were reasonably clean and 13% were not very clean or maintained. Half of Grade 4 Afghan students wear uniforms to go to schools. Afghanistan seems to have made some progress in terms of having access to inputs for teaching, as approximately 86% of Grade 4 Afghan students have textbooks available in the classroom and almost three-quarters of teachers had their lesson plan ready and available. STUDENT SUPPORT Afghan students do not have the necessary support from their home and from the school to allow them to fully be prepared for learning. Despite the severe lack of an enabling environment, these students continue to show resilience and are enthusiastic about going to school, as observed by their attendance and engagement levels in school. A supportive environment can be instrumental in allowing students to focus on their responsibilities at school and be more dedicated towards learning. A stable, enabling environment at home, educated surroundings, teachers who motivate their students, availability of basic equipment like pencils and exercise books, all ensure that students are able to focus more on learning and less on the obstacles in the way of learning (WDR 2018). 3 Do Afghan students receive the necessary support from school? A supportive environment at school offers resources that facilitate learning. Students cannot be expected to learn if they do not have the necessary equipment with which to study. In Afghanistan, only about one-third (36%) of schools have the minimum equipment required in classroom to allow students to learn, which includes having a functional blackboard, an exercise book and pens or pencils. This is considerably low even when compared to Sub-Saharan African (SSA) countries, where the figure is closer to 60%. Surprisingly, rural schools are much better equipped with minimum school equipment compared 3 It is important to point out that the Afghanistan SABER SD could not cover the cost of a full survey of student household conditions and therefore the data for this section are sparser and the discussion more tentative. 8 | The Learning Crisis in Afghanistan to urban schools. There is no statistically significant difference in minimum equipment availability between schools in terms of gender mix. Schools with special classes for students with special needs are substantially scarce in Afghanistan. A third of the Afghan schools (31%) report having special needs students. However, only 2% of the schools offer special classes for students with special needs. Moreover, only 18% of Afghan schools report offering services for students to cope with Post-Traumatic Stress Disorder (PTSD). Urban schools and same-sex schools are better equipped with special needs facilities and PTSD support than rural schools and co-educational schools. Lastly, only half of the teachers provide a supportive environment to students in the classroom in terms of inviting the student to the blackboard, checking their performance individually or calling the students by their names. Do Afghan students receive the necessary support from home? A supportive environment at home also narrows down the responsibilities of students to learning only. Students in Afghanistan do not have a strong support system provided by their home and families. More specifically, less than half (40%) of Grade 4 students in Afghanistan have home support available. Even though almost all students (90%) are provided with breakfast and have missed less than 5 days of school per month, only 50% of students count on someone to help them out with homework. This is reflected in the fact that only a small percentage of students have parents with some level of education. Basically, half of the students reported that their father is illiterate, and around 75% reported that their mother is illiterate. Family support in Afghanistan does not vary significantly by urban or rural areas, or by type of school in terms of gender-mix. Are Afghan students engaged with learning in the classroom? Being present at school and engaged in the classroom paint a picture of the students’ dedication to learning. Around a quarter of the students were observed to be absent on average. Once in the classroom, Afghan student do not tend to misbehave and they seem to be engaged most of the time. WHAT EXPLAINS THESE RESULTS? The provision of education in many low-income countries, including Afghanistan, is characterized by a combination of centralized, but typically weak, state control and often low-capacity, locally governed institutions. It is easy to see how a vicious circle is created in which today’s teachers and principals have gone through an education system that does not adequately prepare them, through a training system with low entry requirements and does not compensate for the flaws in the education system, or through no training at all, and sent into a school where they struggle to teach the next generation of students. At the same time, the institutional incentives for high teacher and principal performance are largely missing, with both career progression and financial rewards delinked from performance. Teacher and principal salaries and promotions are largely determined by seniority and educational qualifications, unrelated to effort or performance. Finally, the various state and local authorities provide limited technical support or supervision. While teachers have autonomy to choose what and how to teach, they do not receive support in terms of material provision or in terms of good professional development or coaching by the school principal and experienced teachers. In light of this evidence, it is no surprise to see the results we present in this report. Absence, which according to the Results of the Afghanistan SABER Service Delivery Survey | 9 teachers and principals is their main responsibility, is low. However, improving student learning is not seen as a main responsibility, and teachers and principals are not focused on achieving this goal. Box 0.1: Applying the WDR Framework to CBE Schools Community-Based Education (CBE) schools were developed during the 1990s when the public-school system had collapsed and home-based education was the only option for girl’s schooling under the Taliban. The Ministry of Education (MoE) recognizes the community-based approach to education as an alternate pathway for improving access to education. By definition, a CBE school is the MoE outreach school/class, and is jointly established by the MoE, communities and facilitating partners (NGOs), and/or the MoE and the community, in remote, rural and sparsely populated areas (villages) where: (i) a gender appropriate public school for children does not exist, (ii) children live at a walking distance of more than three kilometers from a public school, and (iii) a significant number of children have missed the opportunity of formal education due to insecurity, distance, lack of female teachers or a lack of learning materials and supplies, have crossed the school age, and require accelerated learning opportunities. Because there are more public schools for boys than for girls in rural areas, the CBE schools are usually created for girls. Although policy requires NGOs to establish CBE schools at least three kilometers away from public schools, many NGOs use the fact that there are no girl’s schools as a rationale for setting up CBE classes for girls right next to public schools for boys. A CBE school provides education for children from grades 1 to 9 and is an integral part of general education in the country. How do CBE school students differ from public school students in learning outcomes? The main finding is that students in CBE schools tend to significantly outperform those in public schools, both in Language and Mathematics. Despite this good news, the overall learning performance for CBE schools is still worrisome as more than half of Grade 4 CBE students in both Language and Mathematics are also performing significantly below their current grade level. In particular, students in CBE schools not only better master the Grade 1 curriculum, they are especially higher performing in Grade 4 tasks with a higher level of difficulty. In Language, the learning gap is found in more difficult Grade 4 level questions involving the correct use of grammar, tenses, and reading comprehension. Similarly, in Mathematics, the biggest difference is attributed to more difficult questions such as word problems, identifying shapes, understanding fractions and division, and double digit subtraction and multiplication. The public-CBE learning gap in Mathematics is larger than the one in Language and ranges between 15 to 20 percentage points. In line with these results, we also find that CBE Afghan students are, on average, half a curriculum- adjusted year of schooling (0.5 years) ahead of public school students. Moreover, a significantly higher proportion of CBE students are able to perform questions pertaining to Grade 4. 10 | The Learning Crisis in Afghanistan Figure 0.3: Curriculum-adjusted years of schooling by public schools/CBE schools Results of the Afghanistan SABER Service Delivery Survey | 11 What factors are behind the better performance of CBE schools? CBE Schools Public Schools CBE Schools Public Schools Schools Questions 67% teachers ask 33 % teachers ask 30 schools 170 schools higher and lower higher and lower order questions order questions Teachers Positivity 40% teachers create 24 % teachers create 1 teachers per 35 teachers per positive positive school school environment in class environment in class Age Maintenance 13% functioning 46% functioning 28 average age 36 average age toilets toilets Salary Computer 140 USD teacher 100 USD teacher 0% at least 1 40% at least 1 average salary average salary computer computer CAYS Students 2.4 teacher CAYS 3.1 teacher CAYS 28 students per class 41 students per class mathematics mathematics 3 years of 12 years of Multigrade Training experience experience 0% multigrade 6.5% multigrade 33.3% teacher 67.6% teacher classrooms classrooms training training On task Absence 89% teacher time 85.4% teacher time 13% student 29% student absence on task on task absence Lessons 90% teachers 54.5% teachers Water 16.7% safe drinking 80.6% safe drinking introduce and introduce and water water summarizes lessons summarizes lessons Textbooks Wall 14% students with 79% students with 38% boundary wall 77% boundary wall textbooks textbooks Planning 95 % teachers have 85% teachers have lessons planned lessons planned 12 | The Learning Crisis in Afghanistan CBE teachers seem to display significantly higher skills in the classroom compared to public school teachers. While we do not have data to answer this question rigorously, we apply the WDR 2018 framework to understand the degree to which there are differences in the quality of the service delivery for CBE versus Public schools. We find that CBE schools have worse infrastructure and equipment with only 13% of the schools having functioning toilets (versus 46%), 38% having electricity (versus 80%), and only 14% of the students having textbooks (versus 79%). Furthermore, CBE schools have younger teachers (28 vs 36 years old) with less experience (3 vs 12 years), and less formal training. While CBE teachers have higher salaries on average (140 vs 100 USD), they are more prone to experience salary delays (almost 40 percentage points higher). On the positive side, CBE schools have smaller class sizes than Public schools (27 vs 44 students) and lower student absence rate (13% vs 23%). Teacher subject and pedagogic content knowledge cannot explain the difference in CBE students outperforming public school students. Public school teachers outperformed CBE School teachers in Mathematics (3.1 curriculum adjusted years of schooling vs 2.4) and have a similar performance in Language. CBE teachers outperformed Public school teachers in the reading comprehension task (72 vs 44) but underperformed in the Cloze Task (44% vs 32% of the points in the section). Similar results are also observed in pedagogy knowledge. CBE teachers did display significantly higher skills in the classroom. For example, 90% of CBE teachers explained the topic of the lesson at the start and summarized what was learned at the end (versus 54%), and only around 5% of the lesson seemed unplanned to the observers (versus 15%). During their lessons, many more CBE teachers asked questions that required students to recall information or to practice what was learned, asked other questions that required higher order skills, and encouraged students to apply what was learned to different contexts and be creative. 67% of CBE teachers mixed lower and higher order questions in their class (versus 33% Public school teachers). In response to students’ answers, around 40% of CBE teachers consistently gave positive feedback and corrected mistakes without scolding students (versus 24% of Public school teachers). Overall, 3 in 10 CBE teachers (versus less than 2 in 10 Public school teachers) apply the full set of beneficial skills to promote learning—structuring, planning, asking questions, creating a positive environment, and providing constructive feedback—in their lessons. In summary, while we cannot know for certain whether the better skills teachers display in the classroom are behind the relative higher performance of CBE students, these findings provide the rationale for more research on this topic. Results of the Afghanistan SABER Service Delivery Survey | 13 Box 0.2: Priority Areas for Action Based on the findings of the report, below we provide technical policy suggestions. We hope these suggestions provide the starting point for a policy dialogue in which stakeholders discuss how to tailor these solutions to the political realities and local context (WDR 2017 and WDR 2018). 1. Align the goals of the system to learning. In the survey, teachers reported that their main responsibility is being on time to school (86%), which coincides with what principals reported. Only 1% of teachers said that their main responsibility was improve student learning outcomes. The system needs to have a stronger focus on learning, and this message has to come clearly from the leadership team. 2. Increase the amount of time students are effectively taught. Above we discussed that on average students are taught 2 hours and 18 minutes per day, which is well below the OECD benchmark (4hr 30 mins, see Bold et al. 2017) and below the average time students in Sub Saharan African are taught. Dobbie and Fryer (2013) in the context of the US, Bruns et al (2016) in the context of Latin America, and Levy (2015) in the context of PISA participating countries found that increasing the dosage of instruction leads to better student outcomes. 3. Strengthen teacher recruitment, support, and monitoring. In the survey, we found teachers have low content and pedagogical content knowledge. Not only that, they also display poor teacher practices in the classroom. In order to tackle these complex problems, there is a need to strengthen three areas of teacher policies: recruitment, support, and monitoring. On recruitment, there is a need to screen candidates that have a minimum mastery of content knowledge before they go into pre-service. On teacher support, there is a need to help teachers become better at teaching. This involves strengthening both pre-service and in-service training, so teachers solidify their content knowledge and also learn practical pedagogical skills to use in the classroom (e.g. how to check for student understanding or how to provide feedback to students). There is also a need for more support in the school for teachers to improve their practices. Once the system is strengthened to support teachers, a monitoring system is needed to assess which teachers are good performers and which need more help. 4. Strengthen principal recruitment, support, and monitoring. In the survey, we found that principals ignore the critical bottlenecks of their schools and are not trained to support teachers to improve their practice. As in the case of teacher policies, there is a need to strengthen recruitment, support, and monitoring of principals. On recruitment, candidates should be screened to check for teaching and management skills. On principal support, there is a need to help principals become better at using data to guide instruction, observing teachers, and providing them with useful feedback to improve their practice. This involves redefining principals’ role and strengthening both pre-service and in-service training. Once the system is strengthened to support principals, a monitoring system is needed to assess which principals are good performers and which need more help. 5. Improve infrastructure in schools and availability of learning material. In the survey, we found that only one-third (35%) of public schools have minimum infrastructure availability and that almost half (46%) of Afghan schools do not have at least one functioning toilet. We also found out that only 36% of schools had a functional blackboard, exercise books, and pens/pencils. 6. Look for positive deviants and local innovations. In the survey, we found that not only CBE students outperformed public school students, but CBE teachers display better pedagogical skills in the classroom. There is a need to investigate further into the CBE model, as well as to use existing data to identify other positive deviants and learn from them. 14 | The Learning Crisis in Afghanistan Introduction Three decades of conflict and instability have devastated Afghanistan’s social fabric and institutions — corruption, insecurity, weak governance, poor public infrastructure, and the government’s inability to extend rule of law pose challenges to current and future economic growth. Subsequently, 35.8% of Afghans live below the poverty line and GDP has plateaued at 2.5% (real growth rate). 4 Despite some successes in adopting a new constitution and quelling the efforts of the Taliban in the early 2000s, the country has remained one of the most dangerous in the world. Of the 34 million people living in Afghanistan, 1.6 million are internally displaced and nearly 60,000 have fled to Pakistan. Moreover, Afghanistan has the highest infant mortality rate in the world, with 110 babies dying for every 1000 born. For those that do survive, an estimated quarter of children under the age of 5 are underweight. Children who make it to adulthood live to be around 51 years old. In 2017 alone, intensified fighting between the Afghan government, Taliban, and groups claiming allegiance to ISKP resulted in upwards of 2,640 civilian deaths and 5,379 injuries (a slight decrease from the year before). 5 Afghanistan’s education system has felt this acute stress and has attempted to respond appropriately. However, it has been extremely difficult to execute any reforms due to pervasive violence and corruption. For instance, many children, especially girls, have not had access to an education because the Taliban used schools for military purposes and targeted terrorist attacks. Upwards of 1,075 6 schools remain closed due to this violence and instability. Despite these challenges, Afghanistan has achieved a great feat in rapidly expanding enrollment, especially at the primary level. To put this achievement into context, enrollment in Afghanistan has increased 9-fold since 2005. Specifically, enrollment at the primary level consistently grew every year, sometimes by more than 10% annually. Expansion at the lower secondary level has been slightly erratic; with growth rates 4 All facts and figures in this paragraph were taken from: https://www.cia.gov/library/publications/resources/the-world- factbook/geos/af.html 5 https://unama.unmissions.org/civilian-casualties-remain-near-record-high-levels-afghanistan 6 NYT article https://www.nytimes.com/2018/04/01/world/asia/afghanistan-schools- taliban.html?rref=collection%2Ftimestopic%2FAfghanistan&action=click&contentCollection=world®ion=stream&module=str eam_unit&version=latest&contentPlacement=12&pgtype=collection Results of the Afghanistan SABER Service Delivery Survey | 15 ranging from 1-2% to 50% over the course of the last 13 years — these rates have now stabilized at 2- 4%. 7 Unfortunately, upper secondary enrollment has not experienced such levels of growth; however, TVET enrollment has increased substantially since 2002. Afghanistan has had a tough time keeping up with this rapid expansion. The report will show that a large share of children that complete third grade lack basic reading, writing, and arithmetic skills—a state of affairs that is replicated in several low-income countries and that UNESCO (2013) dubbed the “global learning crisis.” To date, there has been no robust, standardized set of indicators to measure the quality of services for students. Existing indicators tend to be fragmented and either focus on final outcomes or inputs, rather than on the underlying systems that help generate outcomes or make use of inputs. In fact, no set of indicators is available to measure the constraints associated with service delivery and frontline providers’ behavior, which have a direct impact on the quality of accessible services for citizens. Without consistent and accurate information on service quality, it is difficult for citizens and politicians to assess how service providers are performing and take corrective action. Furthermore, without detailed data, it is difficult to discern consistent patterns of performance in areas of strength and weakness. This data is crucial to improving education quality and advancing the 2030 SDG agenda. This report discusses an ongoing research program intended to help fill this void. Using data collected through direct observations, unannounced visits, and tests from primary schools in Afghanistan, the report highlights strengthens and weaknesses of the system. In order to do so, we follow the WDR 2018 framework to understand why (or why not) the education system is aligned with learning. 7 Lahire, Nathalie. 2018. Afghanistan: promoting education during times of increased fragility. Washington, D.C.: World Bank Group. 16 | The Learning Crisis in Afghanistan Framework — World Development Report 2018 The framework of the World Development Figure 0.4: Why learning doesn’t happen (WDR 2018) Report 2018: Learning to Realize Education’s Promise (WDR, 2018) provides a solid structure for examining if education systems are aligned with learning. One of the main premises of the WDR 2018 is that schooling (enrollment or attendance) is not the same as learning. The WDR identifies and examines the relationship between four crucial parts of an education system that directly affect student learning: teacher knowledge and motivation, school management, availability of school inputs, and learner preparedness. One of the main messages of the WDR 2018 is that schools are failing learners — not just in Afghanistan but around the world. Struggling education systems often lack one or more of these four key school-level determinants to improve learning. An absence of prepared learners, ineffective teaching, inputs that have nothing to do with learning, and an inability to align these results in system where students do not learn. Many children do not arrive at school ready to learn — if they arrive at all. Malnutrition, low parental investments, and harsh environments associated with poverty severely undermine childhood learning (Lupien and others, 2000; McCoy and others, 2016; Walker and others, 2007). Moreover, many disadvantaged children do not attend school due to pervasive conflict, instability, and financial and Results of the Afghanistan SABER Service Delivery Survey | 17 cultural barriers. Deviating from lower learning trajectories due to deprivation and lack of school participation has long-lasting effects, which further widen gaps in learning outcomes. Teachers are the most important determinant of student learning in both developed and developing countries, though they often lack the skills or motivation needed to be effective (Hanushek, 1992; Rockoff, 2004; Bau and Das, 2017). Despite the importance of teacher quality for student learning, most education systems, including in Afghanistan, do not attract strong candidates to the profession. Furthermore, poor education results in teachers’ lacking basic subject knowledge and pedagogical skills. This translates into poor use of instructional time at the school-level and a system that is ill- equipped to support teachers. School inputs often fail to reach classrooms, and when they do, they seldom affect learning. Devoting enough resources to education is crucial, but often it is how the resources are used that is just as important. Looking across systems and schools, similar levels of resources are often associated with vast differences in learning outcomes (Hanushek, 1995; Mingat and Tan, 1998; Mingat and Tan, 1992; Wolf, 2004). Moreover, increasing inputs in a given setting often has small effects on learning outcomes, which is due to the fact these inputs often fail to make it to where they are intended (Glewwe and others, 2011; Hanushek, 1986; Kremer, 1995). Poor management and governance often undermine schooling quality. Although effective school leadership does not raise student learning directly, it does so indirectly by improving teaching quality and ensuring the effective use of resources (Robinson, Llyod, and Rowe, 2008; Waters, Marzano, and McNulty, 2003). Ineffective school leadership means school principals are not actively involved in helping teachers solve problems, do not provide instructional advice, and do not set goals that prioritize learning. This report builds upon the World Development Report 2018 framework in order to provide a diagnostic that assess the functionality and state of the Afghan education system. 18 | The Learning Crisis in Afghanistan Background of SABER Service Delivery Survey The SABER Service Delivery (SD) tool was developed in 2016 in the Global Engagement and Knowledge Unit of the Education Global Practice (GP) at the World Bank, as an initiative to uncover bottlenecks that inhibit student learning in low and middle-income countries and to better understand the quality of education service delivery in a country as well as gaps in policy implementation. This school survey is aligned to the latest education research on what matters for student learning and how best to measure it. Its main purpose is to provide a mechanism to assess these different determinants of learning through a diagnostic tool and also to uncover the extent to which policies translate into implementation and practice. In alignment with the World Development Report (WDR 2018), the SABER SD instrument examines the four key elements in an education system, which are identified as the main determinants for student learning. This survey strategically collects information on the most important school inputs and processes that produce learning outcomes. The SABER SD survey builds upon and contributes to two current World Bank Group initiatives that produce comparative data and knowledge on education systems: SABER (Systems Approach for Better Education Results) and SDI (Service Delivery Indicators). One of the advantages of the previous surveys, such as SDI, is that we can use them for international comparison reasons. The SABER SD instrument collects data at the school level and asks questions related to the roles of all levels of government (including local and regional). The tool provides comprehensive data on: teacher effort and ability; principal leadership; school governance, management, and finances; community participation; and student performance in Mathematics and Language which includes a classroom observation module. Results of the Afghanistan SABER Service Delivery Survey | 19 Box 0.3: The Link between SABER SD and SDI Approach The foundations of the SABER SD survey build upon two pre-existing World Bank Group initiatives that produce comparative data and knowledge on education systems: SABER (System Approach for Better Education Results) and SDI (Service Delivery Indicators). It also draws upon earlier surveys, namely, QSDS (Quality of Service Delivery Surveys) and PETs (Public Expenditure Tracking). On the one hand, the SABER SD tool builds on the evidence base and captures policy implementation measures from the core SABER domains. 8 On the other hand, the SABER SD tool adapts and extends the surveys that were developed and implemented under the SDI program, which provides a set of metrics for benchmarking service delivery performance in education and health in Africa. The overall objective of SDI is to gauge the quality of service delivery in primary education and basic health services. The SDI approach has been implemented in Kenya, Nigeria, Togo, Uganda, Mozambique, Senegal and Tanzania. 9 There are two main factors that distinguish the SABER SD tool from SDI: (i) it has expanded the measurement of service delivery in primary education outside Africa and into Asia through pilots in Afghanistan, Pakistan and Laos, and (ii) it has adapted and extended the SDI approach by including additional test items from TIMSS and PIRLS, different modality for test administration, different classroom observation modules, and additional questions on student background. All these characteristics of the SABER SD survey make the instrument more appropriate for the Afghan context, while at the same time preserving some international comparability. However, it is important to acknowledge that we do not establish a particular logic behind the international comparisons used for the case of Afghanistan. Basically, the reason behind the choice of countries used for international comparison in this report is based on data availability. 8 The core SABER domains include: Education Management Information Systems (EMIS), Education Resilience, School Autonomy and Accountability, School Finance, Student Assessment, and Teachers. 9 More information on the SDI survey instruments, data and reports, and more generally on the SDI initiative can be found at http://data.worldbank.org/sdi. There are also a series of published papers which have used the data from SDI surveys in order to analyze the learning process in primary schools in Africa, the lost human capital and also teacher’s effort, knowledge and skills such as Bold et al (2017) and Bold et al (2017) -background paper of the WDR 2018-. 20 | The Learning Crisis in Afghanistan Chapter 1 : Is Afghanistan’s Education System Aligned with Learning? This chapter presents the learning outcomes of 3,744 Grade 4 students in public schools, based on data from a Mathematics and Language assessment collected in 2017 by the Afghanistan SABER SD survey. 10 The Afghanistan SABER SD survey was implemented in 200 primary schools across 21 provinces in Afghanistan, of which 170 constitute a nationally representative sample of public schools and the remaining 30 are Community Based Education (CBE) schools managed by NGOs. 11 The Language and Mathematics assessments were administered in the school’s language of instruction — Dari or Pashtu. These tests were administered by enumerators and students were given 30 minutes to complete each subject assessment. These assessments capture the extent to which Grade 4 student’s performance is aligned with the learning structure and curriculum in Afghanistan. This report allows for a closer examination of the real bottlenecks causing student misalignment with an international standard of learning, and thus narrows the focus of the analysis to finding solutions to the corresponding lags in student learning performance. What do Grade 4 Afghan students know? Even though more Afghan students are enrolled in school, they are not learning. Based on the data collected as part of this survey, Afghan student in Grade 4 answered correctly only 30% of the questions on the Language test, on average. In particular, one-third could not identify a picture from a given word, three-quarters could not form a sentence with the verb “went” or the verb “is cooking”, and less than 15% could comprehend a simple paragraph. Their performance in Mathematics is even worse, as Afghan students answered correctly only 25% of the questions on the Mathematics test. 10 Grade 4 students were chosen because: 1) there is no standardized national or international evaluation of this level (students take PASEC in grades two and five only); 2) the sample of children in school becomes more and more self‐selective in higher grades due to drop‐out rates; 3) there is growing evidence cognitive ability is most malleable at younger ages. It is therefore especially important to get a snapshot of student’s learning and the quality of teaching provided at younger ages. 11 Chapter 6 contains detailed information on the learning outcomes of CBE school students and their performance is compared to the public school students. Results of the Afghanistan SABER Service Delivery Survey | 21 Although most Afghan students could add single and double digit numbers and subtract single digit numbers, they were unable to subtract double digit numbers or complete triple digit equations. Both multiplication and division skills have not yet solidified and all but a few students were able to solve word problems, compute fractions, identify shapes, or calculate an area. The key finding in this chapter is that most Afghan students perform significantly below the Mathematics and Language standards set by the Afghan Ministry of Education. Although students are in school, they are not learning much. The curriculum-adjusted years of schooling metric captures how much students learn each year they are in school. Grade 4 Afghan students have on average 1.11 curriculum-adjusted years of schooling in Language and 0.92 curriculum-adjusted years of schooling in Mathematics — meaning that even though Grade 4 students have been in the system for 4 years, they only display the knowledge of a Grade 1 student. More specifically, around 65% of the Grade 4 students have fully mastered the Language curriculum for Grade 1, and only 15% of students could perform Grade 4 level Language questions. The test results in Mathematics are even more concerning as less than half of the students have mastered the Grade 1 Mathematics curriculum. Moreover, less than 3% of Afghan students can solve Grade 4 level Mathematics questions. How does the learning performance vary by type of school? In addition to assessing students’ grade-level knowledge, the SABER SD survey examines learning performance among rural and urban schools, and also same sex and co-educational schools. Of these categorizations, urban schools tend to perform better than rural schools. In terms of gender, four types of schools were observed in Afghanistan as part of this survey: boys schools, girls schools, co- educational schools with same-sex classrooms, e.g. boys and girls sit in different classroom within the same school, and co-educational schools with co-educational classrooms e.g. boys and girls sit in the same classroom. In Language, girls schools tested similarly to boys schools, and performed overwhelmingly better than both types co-educational schools. Boys schools do not differ from girls schools and also from co-educational schools in the school, but they also perform better than co- educational schools in the classroom. Lastly, language performance in co-educational schools is indistinguishable between genders. In Mathematics, Afghan students in same-sex schools have better results than those in co-educational schools. Students at co-educational schools tend to score lowest in both subjects. How do the curriculum-adjusted years of schooling change if: (i) grade qualification criteria is made easier (ii) missing responses are excluded and (ii) students would have tried to guess the test responses? Three robustness-checks were performed to the curriculum-adjusted years of schooling indicator to account for the specific challenges in Afghanistan. First, the percentage of students that qualify for Grade 4 increases slightly. To rule out the theory students did not have enough time to solve the questions at the end of the exam and item non-response bias, the curriculum-adjusted years of schooling results were analyzed using only the questions students attempted. For instance, of the 3,744 students who took the Language test, most of the students (67%) skipped the majority of the questions. Similarly, 43% of students skipped half of the questions on the Mathematics exam. Skipping questions contributes to increasing student’s performance in Language and Mathematics compared to the original estimation given that a higher proportion of students are able to master more difficult 22 | The Learning Crisis in Afghanistan tasks, while a smaller proportion of students are unable to solve Grade 1 tasks. After assigning a randomly selected response to questions skipped by the students, the predicted curriculum-adjusted years of schooling of Afghan students for Language is below what randomness would imply, while the predicted curriculum-adjusted years of schooling for Mathematics remains unchanged. What explains the most variation in the learning performance in Afghanistan? More than one-half (52%) of the variation in learning performance is found across/between schools. Meaning, within the same district or province, there are schools where nearly every student has basic Grade 4 level skills and schools where nearly every student does not. Which is to say, there are good and bad schools in all the districts rather than good schools in certain districts and bad schools in others. Even though school-level variations explain the largest difference in learning outcomes, some differences in student scores are also explained within the same school. For instance, in some of Afghanistan’s top-performing schools, test scores, measured as percentage of correct answers, vary from 20% to 95%. How do Afghan students compare internationally? Overall, Afghan student performance in math is comparable to the average in Sub-Saharan Africa. In particular, the results are on par with Tanzania and Togo. Afghan students significantly outperform students in Mozambique but perform worse than students in Kenya. Moreover, they consistently performed below the TIMSS International average. In Language, Afghan students underperform relative to Pakistan, Botswana, Colombia, and South Africa and are below the Pre-PIRLS international average. This Chapter is organized as follows. The first section assesses Afghan students’ learning performance in Language, while the second section assess Afghan students’ learning performance in Mathematics. The third section examines the curriculum-adjusted years of schooling in both subjects and assess whether Grade 4 students’ performance is aligned with the learning structure and curriculum in Afghanistan as stipulated by the Afghan Ministry of Education. The fourth section analyzes how the learning performance varies by urban/rural schools. The fifth section examines the learning performance by type of school in terms of gender-mix. In the sixth section, we characterize the learning performance beyond the mean, analyzing in particular the learning gap between and within schools. Lastly, we characterize a typical school in Afghanistan and then compare Afghan students’ performance in learning outcomes to other students internationally. SECTION I. STUDENT RESULTS: LANGUAGE ASSESSMENT The Language test assesses students’ understanding of grammar, tenses, vocabulary, and reading comprehension. The questions range from Grade 1 difficulty level, such as asking students to identify a picture from a given word, to Grade 4 difficulty level, which requires them to identify the correct vocabulary term, grammatical form, or tense and complete a sentence or paragraph. Figure 1.1 contains some of the Language questions/items found in the survey. The reading comprehension passage tests students’ ability to retrieve explicitly stated information or draw inferences from a story. This section contains two kinds of questions: (1) those that require students to focus on and retrieve explicitly stated information, e.g. “What did the animals talk about Results of the Afghanistan SABER Service Delivery Survey | 23 every morning?”, which contain the answer directly from the passage in the form of multiple choice: “C. The news of the jungle”, and (2) those that ask students to make straightforward inferences, e.g. “Why didn’t anyone listen to the giraffe?”, which requires students to connect two sentences in the passage and make an inference that the reason for no one listening to the giraffe is attributed to its height and pick the response as “C. He was too tall”. Table 1.1 shows the type of Language questions, their corresponding Grade-level, and level of difficulty. Table 1.1: Language questions and corresponding grade level in Afghanistan Language Questions Corresponding Grade Level in Afghanistan Identify Picture from Word Grade 1 Vocabulary 1 – “farm” Grade 4 - Easy Vocabulary 2 – “grow” Grade 4 - Easy Vocabulary 3 – “father” Grade 4 - Easy Vocabulary 4 – “on” Grade 4 - Easy Grammar 1 – “with” Grade 4 - Easy Grammar 2 – “sheep” Grade 4 - Easy Grammar 5 – “they” Grade 4 - Easy Reading Comprehension 1 – “What did the animals talk about every morning?” Grade 4 - Easy Reading Comprehension 3 – “Which leaves did the giraffe eat?” Grade 4 - Easy Reading Comprehension 4 – “Why were the animals on the ground afraid of the giraffe?” Grade 4 - Easy Reading Comprehension 5 – “What did the giraffe stop doing over the summer?” Grade 4 - Easy Reading Comprehension 9 – “Who told the animals to climb to the treetops?” Grade 4 - Easy Grammar 3 – “behavior” Grade 4 - Intermediate Grammar 4 – “third” Grade 4 - Intermediate Reading Comprehension 2 – “Why didn’t anyone listen to the giraffe?” Grade 4 - Intermediate Reading Comprehension 6 – “Why did the animals huddle together beneath the bushes?” Grade 4 - Intermediate Reading Comprehension 8 – “What made the roaring sound in the distance?” Grade 4 - Intermediate Reading Comprehension 10 –“Why were the animals trying to climb to the treetops?” Grade 4 - Intermediate Reading Comprehension 11 – “Why couldn’t some of the animals climb up the slippery tree Grade 4 - Intermediate trunks?” Tenses 1 – “seek” Grade 4 - Advanced Tenses 2 – “went” Grade 4 - Advanced Tenses 3 – “is cooking” Grade 4 - Advanced Source: SABER Service Delivery - Afghanistan 2017 24 | The Learning Crisis in Afghanistan Figure 1.1: Samples of Language questions Grade 1 Difficulty Level - Identify a picture from a given word Dari Version English Version ‫ ﻟﻄﻔﺂ ﺗﺼﻮﯾﺮ را ﺣﻠﻘﮫ ﻧﻤﺎﯾﯿﺪ ﮐﮫ ﺑﮫ ﮐﻠﻤﮫ ﮔﻔﺘﮫ ﺷﺪه‬.1 Please circle the picture that matches the .‫اﺳﺘﺎد ﻣﻄﺎﺑﻘﺖ داﺷﺘﮫ ﺑﺎﺷﺪ‬ word said by the teacher. Grade 4 Difficulty Level - (Easy, Intermediate and Advanced) Vocabulary sample question Dari Version English Version ‫ﻟﻐﺖ ﺻﺤﯿﺢ را از ﺻﻨﺪوق اﻧﺘﺨﺎب ﻧﻤﺎﯾﯿﺪ و در ﺧﺎﻧﮫ‬ Choose the correct word from the box and .‫ﺧﺎﻟﯽ آﻧﺮا ﺑﺮای ﺗﮑﻤﯿﻞ ﻧﻤﻮدن ﻗﺼﮫ ﺟﺎﺑﺠﺎ ﻧﻤﺎﯾﯿﺪ‬ fill in the blank spaces to complete the story. ‫ ﭘﺪرم‬،‫ ﻣﺰرﻋﮫ‬،‫ در‬،‫ﮐﺸﺖ‬ grow, on, farm, father My parents are farmers. Their _______(1) is ‫( آﻧﮭﺎ ﺑﺴﯿﺎر ﺑﺰرگ‬1)‫ ـــــــــــــــ‬.‫واﻟﺪﯾﻦ ﻣﻦ دھﺎﻗﯿﻦ اﻧﺪ‬ not big. They __________(2) maize and .‫( ﻣﯽ ﮐﻨﻨﺪ‬2)‫ اﻧﮭﺎ ﺟﻮاری و ﭘﻨﺒﮫ ـــــــــــــــــــــــ‬.‫ﻧﯿﺴﺖ‬ cotton. Last year they got very little food. This ‫ ﺑﺨﺎطﺮﯾﮑﮫ‬.‫ آﻧﮭﺎ ﺣﺎﺻﻞ ﺑﺴﯿﺎر ﮐﻢ ﺑﺪﺳﺖ آوردﻧﺪ‬،‫ﺳﺎل ﻗﺒﻠﯽ‬ is because ________(3) became sick and ‫( ﻣﺮﯾﺾ ﺷﺪ و ﺗﻨﮭﺎ ﻣﺎدرم‬3)‫ــــــــــــــــــــــــ‬ mother worked alone _______ (4) the farm. .‫( ﻣﺰرﻋﮫ ﮐﺎر ﮐﺮد‬4)‫ــــــــــــــــــــ‬ Grammar sample question: Dari Version English Version .‫( ﭘﺮوﯾﺰ _________ را ﺑﮫ ﺻﺤﺮا ﺑﺮد‬6) (6) Parviz took the _____ into the desert  ‫ ج( ﮐﺸﺘﯽ‬ ‫ ب( ﮔﻮﺳﻔﻨﺪان‬ ‫ اﻟﻒ( ﮔﻮﺻﻔﻨﺪ‬  a) Sheeps  b) Sheep  c) Ship  d) Ships ‫د( ﮐﺸﺘﯽ ھﺎ‬ Tenses sample question: Dari Version English Version ________ ‫( ز ﮔﮭﻮاره ﺗﺎ ﮔﻮر داﻧﺶ‬10) (10) _______ knowledge from cradle to grave. (‫ د‬ ‫ ج( ﺑﺠﻮﯾﻨﺪ‬ ‫ ب( ﺑﺠﻮی‬ ‫ اﻟﻒ( ﻣﯿﺠﻮﯾﻢ‬  a) Seeks  b) Seek  c) Seeking  d) be ‫ﻣﯿﺠﻮﯾﻨﺪ‬ seeking Results of the Afghanistan SABER Service Delivery Survey | 25 Grade 4 Difficulty Level - Reading Comprehension Passage Dari Version English Version ‫ﺗﻤﺎم ﺣﯿﻮاﻧﺎت‬.‫ﺣﯿﻮاﻧﺎت ﺟﻨﮕﻞ ﯾﮏ ﮔﺮوه ﺧﻮﺷﺤﺎل ﺑﻮدﻧﺪ‬ The Lonely Giraffe (By Peter Blight; Illustrated ‫ھﺮ ﺻﺒﺢ ﺑﺮای ﺑﺤﺚ ﮐﺮدن ﻣﻮﺿﻮﻋﺎت ﺟﻨﮕﻞ ﻣﻼﻗﺎت‬ by Michael Terry) .‫ﻣﯿﮑﺮدﻧﺪ‬ The jungle animals were a friendly bunch. All ‫ اﻣﺎ ھﯿﭻ ﮐﺴﯽ‬،‫ھﺮ ﮐﺪام آﻧﮭﺎ ﺑﮫ ﻧﻮﺑﮫ ﺧﻮد ﺻﺤﺒﺖ ﻣﯿﮑﺮدﻧﺪ‬ the animals met every morning to talk about .‫ﺑﮫ زراﻓﮫ ﮔﻮش ﻧﻤﯿﺪاد‬ the jungle news. ‫ زﻣﺎﻧﯿﮑﮫ زراﻓﮫ ﺳﺮ ﺧﻮد را ﺑﺮای‬.‫زراﻓﮫ ﺑﺴﯿﺎر ﻗﺪ ﺑﻠﻨﺪ ﺑﻮد‬ Everyone took their turn to speak, but no one ‫ﺻﺤﺒﺖ ﮐﺮدن ﭘﺎﯾﯿﻦ ﮐﺮد ﺣﯿﻮاﻧﺎت دﯾﮕﺮ ﻋﻼﻗﻤﻨﺪی ﻧﺴﺒﺖ‬ listened to the giraffe. .‫ﺑﮫ زراﻓﮫ را از دﺳﺖ ﻣﯿﺪادﻧﺪ‬ (13) What did the animals talk about every morning? ‫ ﺣﯿﻮاﻧﺎت ھﺮ ﺻﺒﺢ در ﺑﺎره ﭼﯽ ﺻﺤﺒﺖ ﻣﯿﮑﺮدﻧﺪ؟‬.(13)  A. giraffe ‫در ﺑﺎره زراﻓﮫ‬. A  B. Leopard ‫در ﺑﺎره ﭘﻠﻨﮓ‬.B  C. The news of the jungle ‫در ﺑﺎره ﻣﻮﺿﻮﻋﺎت ﺟﻨﮕﻞ‬.C  D. Monkey ‫درﺑﺎره ﺷﺎدی‬.D Source: This question was taken from the released Pre-PIRLS items (TIMSS & PIRLS, 2011). There are two important remarks on students’ performance on the Language Assessment: (i) it is generally poor, and (ii) it declines considerably with increasing difficulty level. Overall, Afghan students answered approximately 30% or less of the questions correctly in each Language task — indicating most exam questions were beyond the ability of most students. Students’ ability to complete simple Language tasks (such as vocabulary, grammar and tenses questions) is greater than answering the more complex reading comprehension questions. Figure 1.2 provides a complete breakdown of the Language test results, along with the percentage of students who correctly answered the questions. One-third of Grade 4 students could not identify a picture from a given word, which is a Grade 1 level question. 35% of students could properly answer Grade 4 vocabulary questions. Grammar questions, which are of easy and intermediate Grade 4 level, were correctly solved only by 31% of the students. For tenses questions, which are advanced Grade 4 level, only a quarter (26.4%) gave the correct answer. Students struggled most with the reading comprehension section; the average does not exceed 16%. More precisely, only 15% of students could make straightforward inferences, while 17% could retrieve explicitly stated information. These results clearly pinpoint a great difficulty in reading and comprehending long passages. The Language test 26 | The Learning Crisis in Afghanistan contains 4 potential answers for each multiple-choice question (See Figure 1.2), meaning, 25% of the students would have gotten the correct answers had they randomly selected the answers. This suggests Afghan students’ performance in Language is much lower than what randomness would imply, especially for the hardest tasks. Figure 1.2: Student learning performance in Language – Grade 4 Note: Each specific Language test item represents the percentage of student that responded the question correctly. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 27 There are no major differences in the students’ learning performance in Language by type of instruction language (Dari vs Pashtu). In Afghanistan, there are two main languages of instruction: Dari and Pashtu, depending on the region. 12 From Figure 1.3, we can imply that there are not significant differences in student learning performance by type of instruction language. The only difference worth mentioning is that a significantly higher proportion of Pashtun students (28%) compared to Dari students are able to identify a picture from a word, the easiest task of the test. However, Dari students are performing better in other Grade 4 easy level questions. 12 Dari and Pashtu, the two official languages of Afghanistan, are the most widely spoken. Dari (Afghan Persian) is spoken by 50% of the population and serves as the lingua franca in Kabul, Herat, Mazar-i-Sharif and other cities in northern and north-western Afghanistan, mostly in Tajik and Hazara areas, while Pashtu is spoken by 35%, mostly in the South-Eastern Pashtun areas of the country where ethnic Pashtuns are the majority. Currently, textbooks for all grades are developed in both Dari and Pashtu languages. Students study either Dari or Pashtu language depending on the region’s major language. 28 | The Learning Crisis in Afghanistan Figure 1.3: Student learning performance in language by type of language (Dari vs Pashtu) – Grade 4 Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 29 SECTION II. STUDENT RESULTS: MATHEMATICS ASSESSMENT The Mathematics test covers a range of questions that assess students’ knowledge on concepts varying from Grade 1 level skills, such as single digit addition, single digit subtraction, and identifying shapes, to Grade 5 level skills, such as double or triple digit multiplication and solving word problems (Figure 1.4). Most of the questions are Grade 3 or 4 level. Table 1.2 shows the type of Mathematics questions and their corresponding Grade-level and difficulty. Figure 1.4: Samples of Mathematics questions Dari Version English Version :‫ ﻟﻄﻔﺎ ﺟﻮاب درﺳﺖ را ﺑﮫ ﻣﻌﺎدﻟﮫ ھﺎی ذﯾﻞ اراﯾﮫ ﻧﻤﺎﯾﯿﺪ‬.6 6. Please provide the correct answers to the following equations: ‫ﺟﻮاب‬ (‫ﻣﻌﺎدﻻت )ﻣﺴﺎوات‬ ANSWER EQUATION 8+7 8+7 28 + 27 28 + 27 335+ 145 335+ 145 8-5 8-5 57-49 57-49 443-122 443-122 Dari Version English Version :‫ ﻟﻄﻔﺎ ﺟﻮاب درﺳﺖ را ﺑﮫ ﻣﻌﺎدﻟﮫ ھﺎی ذﯾﻞ اراﯾﮫ ﻧﻤﺎﯾﯿﺪ‬.7 7. Please provide the correct answers to the following equations: ‫ﺟﻮاب‬ (‫ﻣﻌﺎدﻻت )ﻣﺴﺎوات‬ EQUATION ANSWER 7 × 8= 7 × 8= 37 × 40 = 37 × 40 = 214 × 104 = 214 × 104 = 6:3 6:3 75 : 5 75 : 5 369 : 3 369 : 3 Dari Version English Version ‫ ﺣﺼﮫ ﮐﺪام ﻣﺴﺘﻄﯿﻞ ذﯾﻞ ﺳﯿﺎه ﺷﺪه؟‬1/3 .9 9. Which rectangle is 1/3 shaded? Answer:____________________ Answer:____________________ 30 | The Learning Crisis in Afghanistan Dari Version English Version ‫ ﻣﺳﺎﺣت ﻣﺳﺗطﯾل ذﯾل ﭼﻧد اﺳت؟‬.14 14. What is the area of the rectangle?  A. ‫ ﺳﺎﻧﺘﯽ ﻣﺘﺮ‬56  A. 56 cm 8 cm 8 cm  B. ‫ ﺳﺎﻧﺘﯽ ﻣﺘﺮ‬160  B. 160 cm  C. ‫ ﺳﺎﻧﺘﯽ ﻣﺘﺮ ﻣﺮﺑﻊ‬56 Sm  C. 56 cm2 Sm  D. ‫ ﺳﺎﻧﺘﯽ ﻣﺘﺮ ﻣﺮﺑﻊ‬160 20 cm  D. 160 cm2 20 cm Dari Version English Version ‫ در اﯾﻦ وﻗﺖ ﺑﮑﺘﺎش ﺑﮫ طﺮف ﺧﺎﻧﮫ ﺧﻮاھﺮ ﺧﻮد ﺣﺮﮐﺖ‬.12 12. This is the time Boktash starts to walk to his .‫ دﻗﯿﻘﮫ ﺑﮫ آﻧﺠﺎ ﻣﯿﺮﺳﺪ‬45 ‫ او در‬.‫ﮐﺮد‬ sister’s house. The walk takes 45 minutes. ‫ﺑﮑﺗﺎش ﭼﮫ وﻗت ﺑﮫ آﻧﺟﺎ رﺳﯾد؟‬ What time did Boktash arrive? 8:25 A.   A. 8:25 6:35 B.   B. 6:35 7:20 C.   C. 7:20 5:40 D.   D. 5:40 ________________ ________________ Dari Version English Version ،‫ اﻓﻐﺎﻧﯽ ﻗﯿﻤﺖ داﺷﺖ‬250 ‫ ھﺮ ﮐﺘﺎﺑﭽﮫ‬،‫ ﮐﺘﺎﺑﭽﮫ ﺧﺮﯾﺪ‬5 ‫ ﺑﺮھﺎن‬.15 15. Bakari bought 5 notebooks, each notebook ‫او در ﻣﺠﻤﻮع ﭼﻨﺪ اﻓﻐﺎﻧﯽ ﭘﺮداﺧﺖ ﻧﻤﻮد؟‬ costs 250 Afghani, how much did he pay in total? _________________:‫ﺟﻮاب‬ Answer: _________________ Results of the Afghanistan SABER Service Delivery Survey | 31 Table 1.2: Mathematics questions and corresponding grade level in Afghanistan Corresponding Grade Mathematics Questions Level in Afghanistan Single Digit Addition Grade 1 Single Digit Subtraction Grade 1 Double Digit Addition Grade 2 Triple Digit Addition Grade 2 Single Digit Multiplication Grade 3 Single Digit Division Grade 3 Double Digit Subtraction Grade 3 Double Digit by Single Digit Division Grade 4 Triple Digit by Single Digit Division Grade 4 Word Problem: Understanding Fractions Grade 4 Word Problem: Division Problem Grade 4 Double Digit Multiplication Grade 5 Triple Digit Multiplication Grade 5 Word Problem: Time Grade 5 Word Problem: Area Calculation Grade 5 Word Problem: Multiplication Word Problem involving Monetary Units Grade 5 Source: SABER Service Delivery - Afghanistan 2017 Students’ performance in Mathematics tasks is substantially lower than what is to be expected from Grade 4 students. Figure 1.5 captures how students performed on each math question, and the percentage of students who answered the question correctly. 77% of students solved a single digit addition problem correctly, and half solved a single digit subtraction problem correctly. Similarly, double digit addition is solved correctly by 49.6% of students, but less than half can do a triple digit addition (27%) or a double or triple digit subtraction (22%) correctly. Both single digit multiplication and division skills have not yet solidified as just 40.7% and 30.9% of students gave correct answers, respectively. They also appeared to struggle with mathematical operations beyond these. More difficult tasks such as double or triple digit division are answered correctly by an even smaller proportion of the class (around 15%). Interestingly, this number drops even further (6-8%) for double or triple digit multiplication tasks. Afghan students performed considerably poor on word problems, even at the easiest difficulty level. For example, only 22% of students correctly identified a square despite learning shapes in Grade 1. Similarly, even though around half of the students understood single digit multiplication and subtraction, when students were given a word problem that tests simple subtraction and division skills, only 15.5% manage to get it right. For instance, an example of a typical word problem is the following: “Ali had 16 apples. He gave away 4 apples. Then Ali divided the remaining apples equally between two 32 | The Learning Crisis in Afghanistan baskets. How many apples did Ali put in each basket?”. More advanced tasks such as multiplication involving monetary units or area calculation were only answered by a handful of students. Figure 1.5: Student learning performance in Mathematics – Grade 4 Note: Each specific Mathematics test item represents the percentage of students that responded the question correctly. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 33 SECTION III. CURRICULUM-ADJUSTED YEARS OF SCHOOLING: METHODOLOGY AND FINDINGS USING ACTUAL QUESTIONS While looking at average scores is a useful measure of student performance, it could be further strengthened by considering Afghanistan’s curriculum. Skills learned earlier in school are reinforced for longer and would therefore have a higher likelihood of having higher response rates. A more useful measure can be created by aligning the curriculum followed in Afghanistan to the questions students were tested on as part of the SABER SD student assessment. This curriculum-adjusted years of schooling estimate maps the concepts tested in the questions to students’ learning grade and therefore shows how student performance is compared to what is expected of them following their own curriculum. Results on the curriculum-adjusted years of schooling revealed big gaps in students’ Language and Mathematics knowledge compared to their expected grade level. In Grade 4, one third of the students have not mastered the Grade 1 curriculum in Language, while more than half of the students have not mastered the Grade 1 curriculum in Mathematics. Figure 1.6 and Figure 1.7 present the distribution of curriculum-adjusted years of schooling in Language and Mathematics, respectively. 13 Curriculum-adjusted years of schooling in Language For the Language test, the average learning performance of Grade 4 students is around the Grade 1 level (1.11 years of schooling), meaning that the average Grade 4 student in Afghanistan displays the knowledge level of a Grade 1 student in Language. In Grade 4, just half of the tested students have fully mastered the Language curriculum for Grade 1. In addition, The SABER SD test results in Language suggest that Afghan students are performing significantly below their current grade level. Only around 15% of students could perform any Language question pertaining to their grade level (Figure 1.6). Figure 1.6: Distribution of curriculum-adjusted years of schooling – Language Source: SABER Service Delivery - Afghanistan 2017 13 A robustness check using an easier threshold for definitions of curriculum adjusted-years of schooling is given in Appendix B.2. 34 | The Learning Crisis in Afghanistan The Language test consisted of Grade 1 and Grade 4 level questions. The Grade 4 questions ranged between easy, intermediate, and advanced when mapped to Afghanistan’s Language curriculum structure (See Table 1.1). Afghanistan Curriculum Standard SABER SD Language Test Results Grade 1: Grade 1 students are expected to identify pictures 35% of all students were unable to identify a picture from a from given words. word. Around half of the students could do this task but failed to do any more advanced tasks from the Grade 4 level. Grade 4: Grade 4 students are expected to solve easier (e.g. 12.5% of students could only solve easier Grade 4 level filling in the blanks with simple vocabulary words to questions, but nothing more advanced. Only around 3% of complete sentences and paragraphs, retrieve explicitly students managed to do intermediate or advanced Grade 4 stated information from a passage), intermediate, and Language questions. advanced Grade 4 level questions (involving more difficult vocabulary usage, use of tenses, and making straightforward inferences from the reading passage) Curriculum-adjusted years of schooling in Mathematics For the Mathematics test, the average student performance of Grade 4 students is also around the Grade 1 level (0.92 years of schooling). The test results in Mathematics are even more worrisome as not only did the average Grade 4 student in Afghanistan display the knowledge level of a Grade 1 student, but more than half of them were not able to master the Grade 1 curriculum. The SABER SD results in Mathematics show that Afghan students are significantly below their current grade level. Less than 3% of all students classified as Grade 4 level were able to solve their current grade level tasks as stipulated by Afghanistan’s curriculum requirements (Figure 1.7). Results of the Afghanistan SABER Service Delivery Survey | 35 Figure 1.7: Distribution of curriculum-adjusted years of schooling - Mathematics Source: SABER Service Delivery - Afghanistan 2017 The Mathematics assessment tested students on a range of questions between Grade 1 and Grade 5 level of difficulty. Students were required to solve the mathematical functions, while the word problems were presented in multiple-choice question format (See Table 1.2). 36 | The Learning Crisis in Afghanistan Afghanistan Curriculum Standard SABER SD Mathematics Test Results Grade 1: Students are expected to know how to add and Half of the students were unable to carry out single digit subtract single digits. addition (8+7) and single digit subtraction (8-5). These are therefore classified as Grade 0 level students. Around 12% of the students correctly solved both single digit addition and single digit subtraction, but were unable to solve any other grade level questions. Grade 2: Students are expected to know how to do double 50% of students knew how to do double digit addition digit OR triple digit addition, as well as all Grade 1 level (28+27), and only 27% could do triple digit addition questions. (335+145). Overall, only 27% of students managed to reach Grade 2 level by correctly solving either double digit or triple digit addition, as well as all Grade 1 level questions, but they were unable to solve any more advanced mathematical tasks. Grade 3: Students are expected to know how to multiply Scores on these questions highlighted a major gap in single digits (7x8), divide single digits (6/3) and subtract student understanding with less than 5% of all students double digits (57-49). managing to solve all 3 concepts correctly, as well as display knowledge of concepts from previous grades. A breakdown of this shows that 41% of students knew how to do single digit multiplication, 31% single digit division, and 23% double digit subtraction. Only 4% of students are classified as having Grade 3 level understanding, meaning they were correct on all questions up to Grade 3 level, but could not solve any more advanced tasks. Grade 4: Students are expected to additionally know how to A breakdown of the questions shows that 17% of a) divide double or triple digits by a single digit, and b) match students correctly solved the double digit division fraction with a shaded figure or solve a simple division (75/5), and 15% solved the triple digit division (369÷3). problem correctly. (e.g. Ali had 16 apples. He gave away 4 15% of the students identified the correct fraction apples. Then Ali divided the remaining apples equally (Which rectangle is 1/3 shaded?), and 15% were able to between two baskets. How many apples did Ali put in each solve a simple division problem. Overall, however, only basket?) less than 3% of all students classified as Grade 4 level correctly completed tasks up to Grade 4 level by Afghanistan’s own curriculum requirements. Grade 5: Students are expected to solve double (37x40), On the slightly positive note, some students (0.75%) triple digit multiplication (214x104), and word problems managed to reach Grade 5 level understanding by being involving multiplication with monetary units, time problem, able to solve double or triple digit multiplication and 2 and area calculation (e.g. Bakari bought 5 notebooks, each out of 3 word problems. 8% of students were able to notebook costs 250 Afghani, how much did he pay in total?) solve 2 out of three word problems. The learning progression in Mathematics is very low. Sorting the questions by order of difficulty for the students, i.e. from the ones students found easiest to answer to those they found the hardest, it is possible to identify which Mathematics concepts where they were not able to progress. Figure 1.8 shows student performance in Mathematics by order of difficulty for the students and also their performance by the accumulated number of correct answers. The blue bars represent the proportion of Grade 4 students that were able to solve a particular task (ordered by level of difficulty), while the green bars represent the proportion of Grade 4 students that were able to progress from the easiest task to a more advanced task. For example, 77% of students correctly solved single digit addition, and Results of the Afghanistan SABER Service Delivery Survey | 37 52% were able to subtract single digits. Collectively, 47% of students were able to solve both these questions correctly. After increasing the level of difficulty, only about 11.3% of all students in Afghanistan were able to solve the five easiest questions of the exam, i.e. single digit addition and subtraction, double digit addition, single digit multiplication and division. Lastly, analyzing the data in terms of Afghanistan’s curriculum standard for Grade 4 level, the data indicate that less than 3% of students could progress correctly in all the tasks that correspond to their current grade and also those for the previous grades, which is remarkably low. 38 | The Learning Crisis in Afghanistan Figure 1.8: Learning progression in Mathematics concepts Note: This Figure shows student performance in Mathematics by order of difficulty for the students and also their performance by the accumulated number of correct answers. The light blue bars represent the proportion of Grade 4 students that were able to solve a particular task, while the dark blue bars represent the proportion of Grade 4 students that were able to progress from the easiest task to a more advanced task (e.g. from 1 to 3 means that the student was able to solve single digit addition, single digit subtraction and double digit addition. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 39 SECTION IV. DISAGGREGATION BY URBAN VS. RURAL SCHOOLS Looking at the disaggregation between urban and rural schools, the general trend is in favor of urban schools in both Mathematics and Language, with a few minor exceptions. In Language, the rural schools had higher average scores in Grade 1 level task of identifying picture from word, but this difference is not statistically significant. However, for most of the other Grade 4 level questions on the Language test, urban schools showed significantly better results than rural schools (Figure 1.9). 14 In Mathematics, urban schools usually either outperformed rural schools or showed similar results to rural schools. In particular, Afghan students in urban schools perform significantly better on word problems and some other relatively difficult tasks such as double and triple digit subtraction (Figure 1.10).15 14 Table C.2 in Appendix C reports the difference in means and p-value in Language questions between schools located in urban and rural areas. 15 Table C.3 in Appendix C reports the difference in means and p-value in Mathematics questions between schools located in urban and rural areas. 40 | The Learning Crisis in Afghanistan Figure 1.9: Students’ learning performance in Language by urban/rural schools Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 41 Figure 1.10: Students’ learning performance in Mathematics by urban/rural schools Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 42 | The Learning Crisis in Afghanistan SECTION V. DISAGGREGATION BY GENDER-MIX: BOYS, GIRLS, CO-ED SCHOOLS, CO-ED CLASSROOM SCHOOLS) Most of Afghanistan’s schools are segregated by gender, with boys and girls studying separately. Four types of schools were observed in Afghanistan as part of this survey: boys schools; girls schools; co-ed schools with same-sex classrooms, e.g. boys and girls sit in different classroom within the same school: co-ed schools with co-ed classrooms e.g. boys and girls sit in the same classroom. Even when girls and boys study separately, the shortage of schools often results in girls and boys attending separate shifts at the same school. Table 1.3 shows more than 80% of the schools are segregated either at the school or classroom level. Only 15% of the schools are co-ed at the classroom level. Co-education schools at the classroom level are much more likely to be in rural areas. Table 1.3: Number of schools by type of school and urban/rural School Category Urban Rural Total % Boys School 17 25 42 25% Girls School 22 29 51 30% Co-Education (class segregation) 28 23 51 30% Co-Education (class mixed) 7 19 26 15% Total 74 96 170 Source: SABER Service Delivery - Afghanistan 2017 Table 1.4 and Table 1.5 show student performance in Language and Mathematics disaggregated by school type, respectively. Results of the Afghanistan SABER Service Delivery Survey | 43 Table 1.4: Student learning performance in language by type of school – Grade 4 Boys Girls COED COED Schools Schools Schools Classroom Statistical (B) (G) (C1) (C2) Difference Identify picture from word 68 59 67 64 Proportion score for 4 vocabulary 33 43 35 27 G>C2 questions Vocabulary1-"farm" 33 39 33 28 Vocabulary2-"grow" 33 43 37 29 G>C2 Vocabulary3-"father" 35 46 37 29 G>C2 Vocabulary4-"on" 31 42 33 23 G>C2 Proportion score for 5 grammar questions 30 40 27 27 G> (C1=C2) Grammar1-"with" 32 43 28 27 G> (C1=C2) Grammar2-"sheep" 31 45 31 29 G> (C1=C2) Grammar3-"behavior" 34 44 30 27 G> (C1=C2) Grammar4-"third" 27 35 23 27 G>C1 Grammar5-"they" 23 36 21 23 G> (C1=C2) Proportion score for 3 tenses questions 28 35 21 22 G> (C1=C2) Tenses1-"seek" 35 41 26 29 G >(C1=C2) ; B>C1 Tenses2-"went" 25 41 23 19 G> (C1=C2) Tenses3-"is cooking" 23 22 16 17 (B=G)>C1 Reading comprehension 18 20 12 13 G >(C1=C2) ; B>C1 Retrieve information 18 21 13 14 (G=B)>C1 Inference 17 20 12 12 (G=B)>(C1=C2) Source: SABER Service Delivery - Afghanistan 2017 In Language, girls schools tested similarly to boys schools, and both performed overwhelmingly better than both types co-educational schools. Co-educational schools in the classrooms tended to score the lowest. Boys schools do not differ from girls schools or school-level co-educational schools, but they do perform better than classroom-level co-educational schools. Lastly, language performance in co-educational schools is indistinguishable between genders. Even though same-sex schools seem to achieve better results in Language than co-educational schools, still less than 40% of students were able to recognize words and sentences as meaningful units of expression and figures as graphical units of expression. Surprisingly, girls schools underperformed compared to the rest of the schools in the easiest task which was to identify a picture from a word. 44 | The Learning Crisis in Afghanistan Table 1.5: Student learning performance in mathematics by type of school – Grade 4 Boys Girls COED COED Statistical Schools Schools Schools Classroom Difference Single digit addition 79 79 80 64 (B=G=C1) > C2 Double digit addition 59 52 45 39 B>(C1=C2); G>C2 Triple digit addition 36 30 25 12 B>C1>C2; G>C2 Single digit subtraction 52 63 49 42 G> (C1=C2) Double digit subtraction 25 26 20 17 (B=G) > C2 Triple digit subtraction 27 29 22 9 (B=G=C1) > C2 Single digit multiplication 47 47 37 28 (B=G) > (C1=C2) Double digit multiplication 10 10 8 6 Triple digit multiplication 9 6 5 3 B> (C1=C2) Single digit division 38 31 28 25 B> (C1=C2) Double digit division 22 15 14 14 B>C1 Triple digit division 17 16 13 13 Understand fractions 18 15 10 16 B>C1 Identify a square 26 27 16 20 (B=G)>C1 Solve a division word problem 15 22 14 9 G>C1>C2; B>C2 Multiplication involving 10 9 3 9 (B=G)>C1 monetary units Time problem 10 17 11 13 G>C1 Area calculation 11 8 4 8 B>C1 Source: SABER Service Delivery - Afghanistan 2017 In Mathematics, Afghan students in same-sex school tend to perform better than those in co- educational schools. In particular, boys and girls schools perform slightly better than the co-ed schools in lower-level tasks, like single digit tasks. However, boys schools outperform girls schools as the level of difficulty in Mathematics tasks increases (e.g. solving triple digit math tasks, understanding fractions, solving multiplication problems with monetary units, and calculating area). SECTION VI. LOOKING BEYOND THE MEAN Difference in Learning Outcomes Between and Within Schools This section aims to analyze differences in Mathematics and Language test scores across/between schools and also within a same school. In other words, we want to compare the performance of students that go to different schools and the performance of students that go to the same school. To examine this, we first rank the schools by average performance in Language and Mathematics and then we plot the students’ overall test scores, school by school, for three different groups of the Results of the Afghanistan SABER Service Delivery Survey | 45 distribution: (i) the five worst performance schools (ii) the five median performance schools and (iii) the five best performance schools, as is shown in Figures 1.11 and 1.12 for Mathematics and Language results, respectively. These Figures also show the full distribution of students’ test scores grouped by the school they attend, which allows us to compare students inside a same school. The horizontal lines indicate students’ knowledge scores (% of total correct answers). Most of the variation in learning is found across schools. Figure 1.11 and Figure 1.12 plot each student’s Mathematics and Language score for the five bottom schools, five median schools and five top schools against a unique school identifier on bottom of the graph. At a first glance, we can clearly observe that some schools are performing fairly well and others are not. For instance, in the worst performance schools, students correctly answered less than 10% of the questions in Mathematics and Language, in the median performance schools, around 20%, while around 75% in top performance schools. This large variation across schools implies that in the same district, there are schools where every student can at least add or identify a picture from a word, and schools where the vast majority of children cannot. Figure 1.11: Differences in learning outcomes between and within schools — Mathematics Note: The Figure plots each student’s Mathematics score against a school identifier. This shows the full distribution of students’ test scores (i.e. percentage of total correct answers) for each school grouped by the school they attended. Source: SABER Service Delivery - Afghanistan 2017 46 | The Learning Crisis in Afghanistan Figure 1.12: Differences in learning outcomes between and within schools — Language Note: The Figure plots each student’s Language score against a school identifier. This shows the full distribution of students’ test scores (i.e. % percentage of total correct answers) for each school grouped by the school they attended. Source: SABER Service Delivery - Afghanistan 2017 Even within the same schools (classrooms) taught by the same teachers, Afghan students are at very different levels of learning. For instance, students in the top 5 performance schools scored around 70% on average in Mathematics, however the similarities seem to end there (Figure 1.11). Schools 23 and 81 have a denser distribution of scores which means that they have students who are performing at roughly similar levels and reasonably well, while schools 160 and 161 have a more varied distribution among their students which means that students in the same class and taught by the same teacher are very different in what they know. 16 Even though students in schools 160 and 161 are top performers, they display a wide spectrum of learning in the same class, ranging from students who cannot perform the most basic Mathematics tasks to those who are close to performing word problems. Similarly, students in the top performance schools in Language also present a large variation in student test scores within the same school (Figure 1.12). For instance, school 98 baffles — even though it is a top performing school in Language, at the bottom of the test score distribution, students end up performing as low as 25% in Language, while the highest performing students go up to an almost full-score. Given 16 All five schools had a comparable number of students, ranging between 18 to 25, so the differences cannot be accounted by a difference in the strength of the tested students. Results of the Afghanistan SABER Service Delivery Survey | 47 this level of variation within the same school, it is very difficult for teachers to respond to the needs of students who cannot even identify a picture from those who can solve a reading comprehension exercise. The same pattern can also be found in the median performing schools, where in the same school (e.g. school 120 in Figure 1.12) there is a relatively high proportion of students who can answer less than 20% of the questions correctly, but there are also students who can be compared to those in the top performance schools. This level of variation in the scoring patterns of students in Afghanistan suggests that some of the differences in student scores are also explained within/inside the same school. Results from the Learning Decomposition This section aims to identify how much of the variability in learning/test score outcomes (i.e. percentage of correct answers in Mathematics and Language) is explained by difference across provinces, districts, or schools. For this purpose, we regress the students’ test scores separately on province, district, and school level dummies to provide an estimate of the explanatory power of each one of these variables. The residual variation is assumed to be driven by differences across students and unexplained variation in the data such as measurement error, ability, etc. More than one-half (52%) of the variation in learning is between schools. Large variation across schools implies that within the same district/province, there are schools were every student can at least master the basic Grade 4 level tasks, and schools where the vast majority of students cannot. Table 1.8 reports the R-squared from the regressions where indicators of province, district, or school are the explanatory variables. The R-squared measures the percentage of the sample variation in student test scores that is explained by difference across provinces, districts, or schools. Results from Table 1.6 indicate that school-level differences carry the most weight in explaining score variation in Mathematics and Language. More specifically, in the SABER SD sample, around half (52%) of the variation in total student test scores is explained by differences across schools rather than across children in the same school. At the subject level, school effects explain around 45% of the variation in Mathematics test scores and 49% of the variation in Language test scores. On the other hand, the portion of variation attributable to provinces and districts in Afghanistan in relatively small. Province variation explains around 20% of the variation in student test scores, and district variation explains only 6% of the variation, which is much lower than the variation at the school level. Table 1.6: R-Squared- Variance decomposition of student’s learning scores in Mathematics and Language R-squared from OLS regression where indicators of province, district or school are the explanatory variables Province District School Average Math score including all questions 0.17 0.05 0.45 Average Language score including all questions 0.18 0.06 0.49 Average Total score including all questions 0.21 0.06 0.52 Source: SABER Service Delivery - Afghanistan 2017 48 | The Learning Crisis in Afghanistan Box 1.1: What does a typical school in Afghanistan look like? Table 1.7 reports the mean, the median, the 10th percentile and the 90th percentile for a number of indicators related to school size, student-teacher ratios and travelling time for students and teachers to get to the school. The typical school in the SABER SD Afghanistan sample has 310 students, 22 teachers, and 16 classrooms. Some schools have as many as 500 to 600 students, especially in urban areas. The median student-teacher ratio is 39:1 and the observed average student-teacher ratio is 41:1, which is near the official benchmark figure of 40:1 suggested by the Afghan government. Nevertheless, the student- teacher ratio is a real problem in schools in the 90th percentile. The student-teacher ratio for the top 10th percentile of schools is 22:1, compared to as high as 68 for the 90th percentile. This number can be as high as 85:1 for urban schools, which suggests that some classrooms are very overcrowded (more information on this below). Afghan students reside around 15 minutes away from their school, which can be even less in urban areas and at most 30 minutes in rural areas. Teachers, on the other hand, travel a little longer. In general, it takes teachers half an hour to get to school, which can range from 15 minutes for urban schools to almost an hour for the average rural schools. However, rural teachers located in the bottom 10 percent of the distribution may need up to 2 hours to get to a school. Table 1.7: Descriptive statistics of schools — Afghanistan 2017 Mean Median 10th 90th percentile percentile Number of Students 356 310 220 563 Number of Teachers 25 22 7 48 Number of Classrooms 21 16 6 47 Student - Teacher Ratio 43 39 22 68 Observed Student - Teacher Ratio 43 41 23 64 Teacher’s Travelling Time (in hours) 0.63 0.50 0.17 1.42 Student’s Travelling Time (in hours) 0.30 0.25 0.08 0.50 Source: SABER Service Delivery - Afghanistan 2017 Table 1.8 shows the percentage of students attending school by type of school in terms of gender. Most of the students in Afghanistan attend co-educational schools where the instruction is segregated at the classroom level, followed by boys schools, then girls schools, and lastly co-educational schools with mixed- gender instruction in the classroom. This last type of school is significantly more likely to be present in rural areas. Results of the Afghanistan SABER Service Delivery Survey | 49 Table 1.8: Average school type and school facilities – Afghanistan 2017 All Urban Rural Students by School Type % students in boy’s schools 29.18 29.36 28.87 % students in girl’s schools 26.03 28.51 21.68 % students in co-ed schools (school level) 35.14 37.24 31.44 % students in co-ed schools (classroom level) 9.64 4.89 18.01 Source: SABER Service Delivery - Afghanistan 2017 Student-Teacher Ratios in Afghanistan Figure 1.13 and Figure 1.14 presents the distribution of student-teacher ratios and observed student- teacher ratios across urban and rural schools using a smooth histogram. The student-teacher ratio is calculated using the enrollment rates at the classroom level, while the observed student-teacher ratio is the one assessed using the classroom observation tool. The Y-axis represents the share of schools for a given student-teacher ratio, while the X-axis represents the student-teacher ratio/observed student- teacher ratio. The typical rural school has a lower student-teacher ratio (33:1) compared to the typical urban school (47:1). The same pattern is sustained for the observed student-teacher ratio. The difference in student-teacher ratio between the top 10% and the bottom 10% in rural schools is smaller (20 students per teacher compared to 61 students per teacher) relative to that in urban schools (25 students per teacher in the top 10% and 85 students per teacher in the bottom 10%). Not only do urban schools have a higher average student-teacher ratio, but there are also many more urban schools with very high student-teacher ratios. Figure 1.13: Distribution of student-teacher ratios – Afghanistan 2017 50 | The Learning Crisis in Afghanistan Figure 1.14: Distribution of observed student-teacher ratios – Afghanistan 2017 Source: SABER Service Delivery - Afghanistan 2017 Girls School and Boys School in Afghanistan 2017 Girls School in Afghanistan. Photo Credit: Anahita Matin/World Bank Results of the Afghanistan SABER Service Delivery Survey | 51 Students at the Abdul Hadi Dawi School getting ready for class. Photo Credit: Anahita Matin/World Bank 52 | The Learning Crisis in Afghanistan Box 1.2: The Learning Crisis: Comparing Afghanistan Students Internationally Overall, Afghan student performance in Mathematics was comparable to the average score of Sub- Saharan Africa, in particular to that of students in Tanzania and Togo. They significantly outperformed students in Mozambique but could not reach the results observed in the TIMSS International average and particularly those in Kenya. (See below for the countries included in TIMSS international average) In order to compare Afghanistan’s Grade 4 students to those from other countries, a specific item breakdown of the results for each subject was conducted. Table 1.9 shows the results for several Mathematics items corresponding to different Grade levels along with the percentage of students who answered the question correctly for each country. Figure 1.15 graphically presents the results for Mathematics items corresponding mostly to Grade 4 level. In single digit addition, 77% of Afghan students responded correctly, which is a relatively similar performance compared to the SSA average (73%), ahead of Mozambique, and behind Kenyan students. In single digit and double digit subtraction, Afghan students scored slightly lower than the SSA average. The only country to significantly underperform Afghanistan was Mozambique at only 18%. Similar double digit questions tested as part of the TIMSS (including Kenya, Tanzania, Indonesia, Bahrain among others) revealed that the international average of these countries (71%) is significantly higher than Afghanistan’s (50%). Understanding of multiplication concepts were significantly higher in Afghanistan at 41% and 9% for single and double digit multiplication respectively, while the SSA average was only 25% and 5% respectively. In Mathematics, tasks that were related to their current grade level (Grade 4) such as single and double digit division, Afghan students also performed quite similar compared to the SSA average, but always better than Mozambique. However, their performance in understanding fractions as part of a word problem was only 15%, which is significantly lower than the TIMSS international average (64%) and even lower than the lowest performing TIMSS country, Morocco at 33%. Table 1.9: Comparing Afghan students internationally in Mathematics – Grade 4 Sub-Saharan Student Overall Score Africa (Out of 100) Afghanistan (Average- All) Mozambique Tanzania Togo Kenya Addition (single digits) Grade 1 77 73 48 78 77 92 Addition (double digits) Grade 2 49.6 53 18 60 65 84 Subtraction (single digits) Grade 1 52.2 63 28 73 65 87 Subtraction (double digits) Grade 3 22.5 29 5 38 22 62 Multiplication Grade 3 40.7 25 4 37 11 51 (single digits) Multiplication Grade 4 8.8 5 0.1 12 6 8 (double digits) Division (single digits) Grade 4 30.9 34 9 38 36 60 Division (double digits) Grade 4 16.6 16 3 21 12 36 Number of Observations 3,431 n/a 1,731 4,041 1,518 2,953 Note: The table presents scores on Mathematics tasks for Grade 4 students in public schools. The average for all SSA countries, reported under the heading “All,” is taken by averaging over the SSA country averages. Source: SABER Service Delivery Results of the Afghanistan SABER Service Delivery Survey | 53 Figure 1.15 : Comparing Afghanistan students internationally in Mathematics – Grade 4 Note: For the double digit addition, TIMSS uses 47+25 while in Afghanistan and Sub-Sahara Africa (SSA) use 28+27. For the Fractions exercise, TIMSS, SSA and Afghanistan use the multiple-choice questions “Which rectangle is 1/3 shaded?”. For the single digit division, Afghanistan and SSA use 6:3. The SSA average includes Mozambique, Tanzania, Togo and Kenya. The TIMSS international average covers 57 countries including some distinct educational systems within countries that have always participated separately throughout IEA’s long history (e.g., the Dutch-speaking part of Belgium and Hong Kong Special Administrative Region (SAR) of the People’s Republic of China). Source: SABER Service Delivery - Afghanistan 2017 54 | The Learning Crisis in Afghanistan Overall, Afghan students performed significantly poorly in Language relative to Pakistan, the international average of Pre-PIRLS countries, and some of the benchmark countries such as Botswana, Colombia and South Africa. Figure 1.16 shows the several Language items corresponding to different Grade levels along with the percentage of students who answered the question correctly for each country. Cross-country comparisons are harder to make for Language-based tests, however in a similarly conducted LEAPS test in Pakistan, students identified picture from a given word 73% of the time, compared to 65% in Afghanistan. The Reading Comprehension exercise was translated from the released Pre-PIRLS items (TIMSS & PIRLS, 2011). One question from the exercise was used for comparison with other Pre-PIRLS countries, where Afghanistan underperformed the Pre-PIRLS International Average, Botswana, Colombia and South Africa, and Afghan students scored only 22% in the same question. The international average for the overall reading comprehension exercise is around 46%, while Afghanistan students scored only around 16%. 17 Figure 1.16 : Comparing Afghanistan students internationally in Language – Grade 4 Note: The reading comprehension complete exercise is “The Lonely Giraffe” both in Pre-PIRLS and Afghanistan. Source: SABER Service Delivery - Afghanistan 2017 17 The TIMSS international average includes the following countries: Armenia, Australia, Bahrain, Belgium (Flemish), Botswana, Bulgaria, Canada, Chile, Chinese Taipei, Croatia, Cyprus, Czech Republic, Denmark, Egypt, England, Finland, France, Georgia, Germany, Hong Kong SAR, Hungary, Indonesia, Islamic Rep. of Iran, Ireland, Israel, Italy, Japan, Jordan, Kazakhstan, Rep. of Korea, Kuwait, Lebanon, Lithuania, Malaysia, Malta, Morocco, Netherlands, New Zealand, Northern Ireland, Norway, Oman, Poland, Portugal, Qatar, Russian Federation, Saudi Arabia, Serbia, Singapore, Slovak Republic, Slovenia, South Africa, Spain, Sweden, Thailand, Turkey, United Arab Emirates and United States. Results of the Afghanistan SABER Service Delivery Survey | 55 Chapter 2 : Teachers The main finding in this chapter is that while teachers in Afghanistan exert high levels of effort as evidenced by their low absence rates and high time on task despite continuous salary delays, they struggle to help student learn as they have low content and pedagogical knowledge and lack teaching skills. For example, almost 20% of teachers were unable to correctly compute a double digit addition and 50% where unable to compute the perimeter of a rectangle. In addition, only 46% could read the Lonely Giraffe story presented in the previous chapter and correctly answer a question about it. This is 5 percentage points lower than the average Grade 4 student from a sample of more than 30 countries that participated in the TIMSS assessment. Poor content knowledge was mirrored by poor pedagogical knowledge and low-quality instruction. On average, less than two in ten used effective teaching strategies to promote learning, such as lesson structuring, planning, asking lower- or higher-order questions and providing feedback. If “adequate” teaching is characterized as being taught by teachers with at least basic pedagogical knowledge and minimum subject knowledge for at least 3 hours a day, then essentially no public primary school in Afghanistan offers adequate quality education. These results point to two systemic problems: the system used to select and train teachers does not deliver high-quality candidates; and the system used to monitor and support teachers does fail to help them to deliver high-quality teaching. Results of the Afghanistan SABER Service Delivery Survey | 57 What do teachers do? 15% of teachers were absent from class, either because they were absent from school or in the school, but not in the classroom. While in the classroom, teachers spent 80 percent of the time on learning activities, close to the gold standard of 85%. Factoring in teacher absence from school and class as well as the percentage of the lesson that the teacher devotes to non-teaching activities, students are taught, on average, 2 hours and 18 minutes per day out of 3 hours and 25 minutes of schedule teaching time. Despite this, the short-scheduled teaching time means that Afghan students received less teaching than the average Sub-Saharan African country where comparable data was collected. This includes Kenya, Mozambique, Togo, Tanzania, Senegal, Uganda, and 4 states in Nigeria. What do teachers know? Teachers were asked to mark (or “grade”) mock student tests in language and in Mathematics. The exercise assessed their ability to spell simple words (“traffic,” for example) and identify the correct grammatical option that completes a sentence such as “[ ] [Who, How much, How many] oranges do you have?”. On average, language teachers correctly answered these questions 75% of the time. They also were asked to correct spelling, grammar, syntax, and punctuation mistakes in a child’s letter that included segments such as “I went to tell you that my new school is better the old one I have a lot of thing to tell you about my new school in Kabul”. Teachers struggle with this and got only 30% of the points. The test also included Cloze passages, which included “student” responses such as “[Where] do I have to go to the market?” (In this case, a correct answer could be either “Why or When.”). Teachers got 44% of the points in this exercise. Finally, they were asked to read a passage from the Lonely Giraffe and answer two reading comprehension questions. The average score for both questions is 44 out of 100 showing teachers lack basic reading comprehension skills we expect 4th students to have mastered. In Mathematics, we tested if the teacher can accurately correct children’s work in such aspects of numeracy as manipulating numbers using whole number operations. In essence, this measure whether the math teacher masters his or her students’ curriculum. Fewer than 40% of Mathematics teachers show evidence of mastering Grade 4 curriculum. Looking at specific tasks in Mathematics, almost 2 in 10 teachers cannot add double digits, a quarter of the teachers cannot do a single digit division and more than one-third of the teachers cannot multiply triple digit numbers. How well do teachers teach? To assess how well teachers teach we measure: (i) teachers’ pedagogical knowledge, (ii) teachers’ capacity to assess students and monitor their progress, (iii) teachers’ actual use of a monitoring system and knowledge of student performance; and (iv) teachers’ classroom practices based on direct lesson observation. To measure pedagogical knowledge, we asked teachers to prepare a lesson plan by reading and extracting information from a factual text on a topic and to state what they would expect their students to learn from the lesson. While teachers struggle to read and understand the text (average score of 39%), they struggled even further to formulate what they wanted children to learn from the lesson based on their reading (average score of 30%). 58 | The Learning Crisis in Afghanistan To measure teachers’ ability to assess students’ learning and give feedback, teachers were asked to: (i) prepare lower- and higher-order questions, (ii) use a marking scheme to give feedback on strengths and weaknesses in students’ writing, and (iii) use a list of students’ grades to turn the raw scores into averages and comment on the students’ learning progression. A few teachers demonstrated ability in at least one of these areas (average score of 25%), and no teacher in the sample could answer at least 80% of the items correctly. To go beyond capacity and measure whether teachers implement a monitoring system we asked teachers: (i) to show us their records of individual students’ performance through the year, and (ii) to estimate their students’ knowledge in three areas of their curriculum (e.g. percentage of students that could answer correctly a single digit addition question). We then compare their answer with the scores from the student assessment module. Less than a third of teachers keep a record of students’ performance and even fewer were able to estimate their performance in those selected areas within a twenty-point margin. No teacher in the sample could estimate all three areas correctly. Poor knowledge of pedagogy was mirrored in behavior in the classroom. Few (less than two in ten) deploy the teaching practices identified in the literature as promoting learning in their lessons —structuring, planning, asking lower and higher order questions and giving feedback. What explains these results? First, the provision of education in many low-income countries, including Afghanistan, is characterized by a combination of centralized, but typically weak, state control and low-capacity, locally governed institutions. It is easy to see how a vicious circle is created: today’s teachers have gone through an ineffective education system that does not adequately prepares them, through a teacher training system with low entry requirements that does not compensate for the flaws in the education system, or through no training at all, to be sent to a school where they struggle to teach the next generation of students. Second, the institutional incentives for high teacher performance are largely missing, with both career progression and financial rewards delinked from performance. Teachers’ salaries and promotions are largely determined by seniority and educational qualifications and are unrelated to effort or performance. Finally, the various state and local authorities provide limited technical support or supervision. While teachers have autonomy to choose what and how to teach, they do not receive support either in terms of materials and good practices or in terms of good professional development or coaching by the school principal or experienced teachers. In light of this evidence, the results we present in this chapter come as no surprise. Absence, which according to the teachers is their main responsibility, is low. However, improving student learning is not seen as a main responsibility, so teachers and schools are not focused on achieving this goal. In the sections that follow, we describe the answers to the questions raised above in more detail: What do Teachers do? (Section I), What do Teachers Know? (Section II), How well do Teachers teach? (Section III), and Why we found these results? (Section IV). Results of the Afghanistan SABER Service Delivery Survey | 59 SECTION I. WHAT DO TEACHERS DO? UNDERSTANDING TEACHERS EFFECTIVE USE OF TIME Being present in the classroom is a conditio sine qua non for teachers to exert effort at teaching. To measure the time teachers spend teaching, an extended approach of that described in Chaudhury et al. (2006) and Bold et. Al. (2017) was employed. In each school, during a first announced visit, up to 10 teachers were randomly selected from the teacher roster. At least two teaching days after the initial survey, an unannounced visit was conducted, during which the enumerators were asked to identify whether the selected teachers were in the school, and if so, if they were in class teaching. Both assessments were based on directly observing the teachers and their whereabouts. Figure 2.1 summarizes the findings and Appendix C provide urban/rural and school type desegregation and statistical significance of the differences between them. Averaging across schools, 15% of teachers were absent from class, either because they were absent from school, or they were in the school but not in the classroom. What do these results imply for the amount of instruction time that students receive? To answer this, the surveys first recorded the length of scheduled teaching time in a day according to school records. Averaged across schools, this comes to 3 hours and 25 minutes. We then multiply this number by the proportion of teachers absent from class. If ten teachers are supposed to teach 3 hours and 25 minutes per day, yet one teacher is absent from either the school or the classroom at any one time, then the scheduled teaching time is reduced to 3 hours and 5 minutes 18. Moreover, even when in the classroom, teachers may not necessarily be teaching. We carried out classroom observation as part of the survey, recording ten snapshots of what the teacher was doing, for a randomly selected fourth-grade Mathematics or language class. The percentage of the lesson lost to non-teaching activities was 20%. We then combine the absence-adjusted teaching time with the proportion of classroom time devoted to actual teaching activities to estimate instruction time as experienced by students. Students are taught, on average, 2 hours and 18 minutes per day, roughly 70% of the schedule teaching time (as shown in Figure 2.1). Table C.6 in the Appendix shows there is similarity across school types (Boys/Girls/Coed School/Coed Classroom) but urban schools tend to have higher percentage of teachers that are in the school but not in the classroom when they are supposed to be teaching. 18 In our case, with 15% absence the scheduled teaching time is reduced to 2 hours and 54 minutes. 60 | The Learning Crisis in Afghanistan Table 2.1 compares Afghanistan’s teachers Figure 2.1: What do teachers do? with teachers from other countries in terms of Absent their absence and effective use of time. To put from school In classroom, teaching this in perspective, Bangladeshi and Indian In school, school absence was measured at 16% and 25% not in 10% classroom 68% respectively in 2003, and classroom absence 5% was estimated to be at 34% in India. Results from Sub-Saharan Africa, Laos, and Latin In classroom, not teaching 17% America show that teachers in these countries are absent from the school between 13-23% of the time and from school or classroom between 26-44% the time. Compared to these results, Afghanistan shows a considerably lower school absence and classroom absence rate. Not only that, but once teachers are in the Note: This figure shows the population average for teachers’ use of time classroom, they are close to the 85% gold for the 170 public schools. It includes 1454 teachers. To compute these numbers, we use a teacher weight to represent the average teacher in standard of time on task. Bruns and Luque Afghanistan. Absence is calculated based on an unannounced visit. (2014), drawing on data from a large sample of classrooms in seven Latin American and the Caribbean countries, show that teachers only spend 52-85% of class time on academic activities (or time on task), implying a loss of potential instructional time equivalent to one day of instruction per week. However, even though Afghanistan teachers are more frequently present in the school or classroom and spend a higher amount of the class time on learning activities, the overall time spent teaching per day is the lowest out of those for which we have data, well below the Sub- Saharan Africa average. Only Mozambique with 1 hour and 43 minutes has lower percentage of teaching time each day. This is explained by the low amount of scheduled teaching time in Afghanistan. Table 2.1: Comparison of teachers’ time use Sub-Sahara Latin Indicator Afghanistan Bangladesh India Africa Laos America School Absence (%) 10 16 25 23 16 13* Classroom Absence (%) 15 34^ 44 26 27^ Time on Task (%) 80 65+ Schedule Teaching Time 3h 25m 5h 31m 5h 18m Time Spent per day Teaching 2h 18m 2h 53m 4h 8m Note: The table reports the school and classroom absence rate for all teachers, time on task, the scheduled teaching time, and actual teaching time. All individual country statistics are calculated using country-specific sampling weights. Teachers are marked as absent from school if during the second unannounced visit, they are not found anywhere on the school premises. Otherwise, they are marked as present. Teachers are marked as absent from class if during the second unannounced visit, they are absent from school or present at school but absent from the classroom. Otherwise, they are marked as present. Time on task is measured using a classroom observation and is computed as the percentage of the lesson lost to non-teaching activities during a lesson. The scheduled teaching time is the length of the school day minus break time. Time spent teaching adjusts the length of the school day by the share of teachers who are present in the classroom, on average, and the time the teacher spends teaching while in the classroom. For the figures for the other countries we use Chaudhury et al 2006, Bold et al 2017 and the SABER SD Laos report (2018). *For Latin America the paper computes Ecuador at 14% and Peru at 11% school absence rate. + For time on task in Latin America we are using data from Bruns and Luque (2014). Because a different classroom observation tool was used in Afghanistan compared to Sub- Saharan Africa and Laos, we do not report the time on task for those countries in the table. ^ Own estimation based on the microdata provided by the authors of Chaudhury et al 2006. Results of the Afghanistan SABER Service Delivery Survey | 61 Box 2.1: Profile of Afghanistan’s Teachers On average, Afghan teachers are 36 years old with the youngest being 17 and the oldest 75. 43% are male (57% female) and less than 1 in 10 teachers in public schools are contract teachers. They have, on average, 14 years of education, with a small number of teachers having only completed primary school, roughly 15% completing Grade 12, 65% completing Grade 14, and 20% completing additional formal education. Teachers have an average of 12 years of experience, but this masks the fact that some of them are new and some have more than 30 years of experience. On average, teachers teach almost 4 hours a day, with 1 hour being the minimum and 10 hours the maximum. The average monthly teacher salary comes to about 100 USD with a dispersion from 10 to 300 USD (including allowances). 40% of the teachers reported receiving their salary with delays. Roughly 6 in 10 teachers work only as a teacher with the other 4 have other jobs with about a quarter of those earning less than 50% of their monthly income out of their teaching salary. While there are some differences by type of school and location (e.g. girls schools have 90% female teachers), for the most part these average across public schools provide an accurate portrait of teachers even at the urban/rural or boys/girl/co-ed school or co-ed classroom level. Table 2.2: How does the average Afghan teacher looks like? Indicator Mean Min Max Age 36 17 75 Male (%) 43 Public School Teachers 91 (%) Years of Education 14 6 18 Years of Experience 12 0 52 Hours per day teaching 3h 48m 1h 10h Monthly salary (in thousands) 7.57 0.67 22.18 Salary Delays (%) 41 Source: SABER Service Delivery - Afghanistan 2017 62 | The Learning Crisis in Afghanistan SECTION II. WHAT DO TEACHERS KNOW? MEASURING TEACHERS’ CONTENT KNOWLEDGE IN LANGUAGE AND MATHEMATICS To measure the subject content knowledge of primary school teachers, and specifically those teaching in the lower primary grades, all Language and Mathematics teachers teaching Grade 4 in the current year (or Grade 3 in the previous year) were assessed. On average, five teachers were tested in each school. In contrast to other approaches to assess teachers’ knowledge, where teachers take exams, teachers here were asked to mark (or “grade”) mock student tests in Language and in Mathematics 19. This method of assessment has two potential advantages. First, it aims to assess teachers in a way that was consistent with their normal activities, namely marking student work. Second, by not testing teachers in the same way as students are tested, it recognizes teachers as professionals. In the analysis, we assess the language knowledge of those teachers who teach Language, and the mathematics knowledge of those teachers who teach Mathematics. All questions on the teacher test were based on common items taken from the primary curricula from Afghanistan. Language (Dari/Pashto) We start by assessing language tasks on the teacher test that covered (roughly) the lower primary curriculum (first through third year of primary school) — specifically, spelling and simple grammar exercises. Almost 75% of teachers made it over this low bar, though with some variation across subgroups as shown in the appendix in Table C.7. Possessing knowledge equivalent to the fourth-grade curriculum is not sufficient to teach language in lower primary as language teaching is “monolithic.” That is to say, teaching a student how to compose even a simple text requires knowledge that goes well beyond what is graded in the curriculum 20. We therefore measure teachers’ ability to correct children’s work in such aspects of literacy as reading comprehension, vocabulary, and formal correctness (grammar, spelling, syntax, and punctuation), all of which are competencies a teacher in lower primary would routinely be required to use. To this end, the language test contained (in addition to the spelling and grammar exercises) items involving sentences with blank spaces where students need to fill in words—so-called Cloze passages—to assess vocabulary and reading comprehension, and a letter written to a friend describing the student’s school, which the teacher had to mark and correct. The performance on this part of the test was alarming, with few teachers mastering the content knowledge in language. Which areas of language teaching are especially problematic? Figure 2.2 offers a breakdown of specific tasks on the language test. First, some teachers are weak in all areas of the curriculum: A quarter of teachers could not correctly answer a grammar exercise that asked them to identify the option, out of three, that would complete a sentence such as “[ ] [Who, How much, How many] oranges do you have?” Second, most teachers struggled with those tasks that required at 19 The subject test was designed by experts in international pedagogy and validated against 13 low income countries’ primary curricula and national teacher standards (Botswana, Ethiopia, Gambia, Kenya, Madagascar, Mauritius, Namibia, Nigeria, Rwanda, Seychelles, South Africa, Tanzania, and Uganda). See Johnson, Cunningham and Dowling (2012) for details. Before using this test in Afghanistan, the team consulted with Afghan curriculum and textbooks to show that it was appropriated and validated it with the government. 20 Because of this we do not have a big enough overlap between the items we asked of students and teachers in the language portion of the test. As a result, we do not present the curriculum adjusted years of schooling for language teachers, but we do this for Mathematics teachers. Results of the Afghanistan SABER Service Delivery Survey | 63 least some knowledge beyond the lower primary curriculum to mark. Less than half of the items in the Cloze passage were marked correctly, which included “student” responses such as “[Where] do I have to go to the market?” (In this case, a correct answer could be either “Why or When.”). Teachers corrected only 30% of the spelling, grammar, syntax, and punctuation mistakes in a child’s letter that included segments such as “I went to tell you that my new school is better the old one I have a lot of thing to tell you about my new school in Kabul.” Finally, teachers were asked to read a passage from the Lonely Giraffe (See chapter 1 for the actual passage) and answer two reading comprehension questions. The average score in both question is 44 out of 100 showing teachers lack basic reading comprehension skills we expect Grade 4 students to have mastered. While teachers in different type of schools and locations were for the most part homogenous, teachers in girls schools were the highest performers and teachers in Co-Ed Classroom were the lowest (see Table C.7 in the appendix). Figure 2.2: Teachers’ content knowledge in Language by type of task Note: This table shows the population average for teachers’ mastery of language content knowledge for the 170 public schools. It includes 1,134 teachers. To compute these figures, we use a teacher weight to represent the average teacher in Afghanistan. Source: SABER Service Delivery - Afghanistan 2017 The result of this analysis is unchanged by looking at teacher knowledge excluding missing responses as for the most part, unlike the student assessment describe in Chapter 1, teachers answered most of the questions. Results are available upon request. Language Test International Comparison Teachers in Afghanistan perform comparably with teachers in Sub Sahara Africa where the same survey was conducted in 7 countries: Kenya (2012), Mozambique (2014), Nigeria (2013), Senegal (2010), Tanzania (2010, 2014), Togo (2013), and Uganda (2013). Teachers are close to the average in all three groups of measures, with a slightly lower score on grammar and slightly higher on the composition task (Table 2.3). 64 | The Learning Crisis in Afghanistan Table 2.3: Comparison of teachers’ performance on Language Language Average SSA (score out of 100) Afghanistan Country Min Max Grammar task 75 79 58 [NIG] 92 [KEN] Cloze task 44 44 27 [TGO] 66 [KEN] Correct composition task 29 26 9 [MOZ] 50 [KEN] Number of teachers 1134 3770 Note: The table presents scores on Language tasks for teachers in public schools teaching grade 4 or who taught grade 3 in the previous year. Language knowledge is computed for teachers teaching language. All individual country statistics are calculated using country-specific sampling weights. The average for all countries, reported under the heading “All,” is taken by averaging over the country averages. The ISO 3-digit alphabetic codes of the countries with the lowest (Min) and highest (Max) score for each item are given in brackets. The figures from Sub Sahara Africa are taken from Bold et al 2017. Source: SABER Service Delivery - Afghanistan 2017 In addition to the comparison with Sub-Saharan Africa we included an item from the TIMSS Grade 4 global student assessment (See Figure 2.3). Figure 2.3: TIMSS Item used as part of the Teacher Language Assessment Results of the Afghanistan SABER Service Delivery Survey | 65 The question tests the teachers’ ability to read a passage and answer a reading comprehension question. 46% of teachers correctly answered the question. This puts teachers in Afghanistan below the average Grade 4 student that took the TIMSS exam, below students in Colombia and South Africa, and just above students in Botswana (Figure 2.4). Figure 2.4: TIMSS Item used as part of the Teacher Language Assessment Note: This table compare the percentage of teachers in Afghanistan and students in TIMSS countries that answer a reading comprehension item correctly. Source: SABER Service Delivery - Afghanistan 2017 Mathematics In Mathematics, we assess teachers’ subject knowledge by checking whether they can accurately correct children’s work in such aspects of numeracy as manipulating numbers using whole number operations. In essence, this measures whether the Mathematics teacher masters his or her students’ curriculum. Fewer than 40% of Mathematics teachers show evidence of mastering Grade 4 curriculum. Looking at specific tasks in Mathematics, nearly 2 in 10 teachers cannot add double digits, a quarter of the teachers cannot do a single division, and more than one-third of teachers cannot multiply triple digit numbers (See Figure 2.5). Of course, we would expect a competent Mathematics teacher to have knowledge beyond that of his or her students; therefore the Mathematics test also included questions one would only encounter in upper primary school. Many Mathematics teachers struggled with these tasks: only 60% of teachers could interpret information in a Venn diagram, and only a third could interpret data from a chart and correctly answer an associated problem (see Figure 2.5 and Figure 2.6). Teachers in urban schools show slightly better results. 66 | The Learning Crisis in Afghanistan Figure 2.5: Teachers’ performance in Mathematics by type of task Note: This table shows the population average for teachers’ mastery of content knowledge for the 170 public schools. It includes 1,598 teachers. To compute these figures, we use a teacher weight to represent the average teacher in Afghanistan. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 67 Figure 2.6: Teachers’ content knowledge item — Peter’s Telephone Bill Note: This is a released item from the TIMSS Grade 4 student assessment. Source: SABER Service Delivery - Afghanistan 2017 As we will see below, this low competence in interpreting data has implications for teachers’ ability to monitor their students’ progress. Finally, teachers struggle to solve more advanced exercises, with less than half able to solve a simple algebraic equation: ( − − = ) Solve for n. In the case of the Mathematics assessment, there is a large overlap of the items tested for students and teachers. As a result, we can compute a similar curriculum-adjusted year of schooling measure as we did in the previous chapter. The Mathematics assessment tested teachers on a range of questions between Grade 1 and Grade 7 level of difficulty. On average, teachers have cumulated 3 years of human capital, with 4% of them having zero years, and only 5% reaching 7 years of human capital (Figure 2.7). 68 | The Learning Crisis in Afghanistan Afghanistan Curriculum Standard SABER SD Mathematics Teachers Test Results Grade 1: We score them as having one year of human capital Around 4% of the teachers were unable to carry out single if they can do single digit addition and single digit digit addition (8+7) AND single digit subtraction (8-5). These subtraction, and unable to do any of the more advanced are therefore classified as Grade 0 level teachers. Around 8% tasks. of the teachers correctly solved both single digit addition and single digit subtraction, and were unable to solve any other grade level questions. These are therefore classified as Grade 1 teachers. Grade 2: We score them as having two years of human Overall around 45% of all Grade 4 teachers tested were at capital if they can perform double digit addition OR triple the Grade 2 level by correctly solving for either double digit digit addition. or triple digit addition, as well as all Grade 1 level questions, and being unable to solve any advanced mathematical tasks. Grade 3: We classify them as having three years of human Scores on these questions highlighted a major gap in teacher capital if they can do double digit subtraction and calculate understanding with less than 4% of all teachers managing to perimeter of a rectangle. master both concepts. Grade 4: We class them as having accumulated four years of Overall, only around 6% of all Grade 4 teachers displayed human capital if they can divide double or triple digits by a knowledge of Grade 4 level skills and could not solve any single digit, and match a fraction with a shaded figure or more advanced tasks. solve a simple division problem. Grade 5: We class them as having five years of human capital Around a quarter of teachers were able to accomplish these if they can multiply double and triple digits and solve 2 out tasks. of 3 word problems such as Bakari bought 5 notebooks, each notebook costs 250 Afghani, how much did he pay in total? correctly. Grade 6: We classify them as having six years of human Only 2.4% of the teachers that solved the previous task could capital if they can interpret Venn diagrams and solve a also solve this one. time/distance problem correctly Grade 7: We classify them as having seven years of human Only 5% of the teachers that solved the previous task capital if they can solve for an unknown in an algebraic correctly could also solve this one. equation Results of the Afghanistan SABER Service Delivery Survey | 69 Figure 2.7: Distribution of curriculum-adjusted years of schooling – Mathematics teachers Source: SABER Service Delivery - Afghanistan 2017 Excluding missing responses does not change the results of these analyses because, unlike the student assessment described in Chapter 1, teachers answered most of the questions. Results are available upon request. 70 | The Learning Crisis in Afghanistan Mathematics Test International Comparison Teachers in Afghanistan perform slightly better than the average teachers in Sub Sahara Africa but well below the best performers (Table 2.4). In addition to the comparison with Sub-Saharan Africa, we included items from the TIMSS Grade 4 global student assessment, and teachers score similarly to the average Grade 4 students that participated in the TIMSS assessment. Table 2.4: Comparison of teachers’ performance on Mathematics Mathematics Average TIMSS Average SSA (score out of 100) Afghanistan Student Teacher Min Max Can add double digits* (%) 82 71 91 75 [TGO] 98 [KEN] Can subtract double digits (%) 90 - 77 59 [NGA] 94 [TZA] Can multiply double digits (%) 78 p 68 44 [MOZ] 89 [SEN] Can solve simple math story problem (%) 67 - 55 17 [MOZ] 91 [SEN] Understands a Venn diagram (%) 59 - 41 19 [TGO] 70 [KEN] Can interpret data in a graph (%) 36 - 25 12 [TGO] 62 [KEN] Can solve algebra (%) 44 - 35 3 [MOZ] 74 [KEN] Fractions (%) 74 64 - - - Telephone bill exercise (%) 33 39 - - - No. of teachers 799 - 3957 Note: The table presents scores on Mathematics tasks for teachers in public schools teaching Grade 4 or who taught Grade 3 in the previous year. Mathematics knowledge is computed only for teachers teaching Mathematics. All individual country statistics are calculated using country-specific sampling weights. The average for all countries, reported under the heading “All,” is taken by averaging over the SSA country averages. The ISO 3-digit alphabetic codes of the countries with the lowest (Min) and highest (Max) score for each item are given in brackets. The figures from Sub Sahara Africa are taken from Bold et al 2017. *TIMSS use 47+25 while in Afghanistan and Sub-Sahara Africa we use 28+27. Source: SABER Service Delivery - Afghanistan 2017 Teachers in Afghanistan struggle reading charts and answering related questions. For example, for the exercise in Figure 2.8, Afghan teachers were outperformed by the average Grade 4 student (33% vs 39%). Not only that, but Grade 4 students in Korea had a 44 percentages point gap (77% vs 33%). Results of the Afghanistan SABER Service Delivery Survey | 71 Figure 2.8: Peter’s (Parvez) Telephone Bill Exercise SECTION III. HOW WELL DO TEACHERS TEACH? TEACHERS’ PEDAGOGIC CONTENT KNOWLEDGE AND TEACHING SKILLS Good teaching is not just about subject content knowledge. It also requires that teachers know how to translate their subject knowledge into effective pedagogy and then apply this in the classroom. Teachers must also know how to assess student capabilities and react appropriately, for example, by asking questions that require various types of responses and by giving feedback on those responses, commonly referred to as “knowledge of the context of learning” (Johnson, 2006; Danielsson, 2007, Pianta et al., 2007, Coe, Aloisi, Higgins and Major, 2014; Ko and Sammons, 2013, Mujis et al., 2014; Vieluf et al., 2012). In a recent review, although not focused on Afghanistan specifically, Mujis et. al. (2014) identify a set of elements that categorize behavior in the classroom that are consistently associated with gains in student learning. They are: i) structuring lessons, and in particular, introducing topics and learning outcomes at the start of the lesson and reviewing them at the end; ii) frequently checking for student understanding by asking questions and allowing time for students to review and practice what they learned, either individually or in groups; iii) varying the cognitive level of questions by mixing lower- and higher-order questions; and iv) creating a positive learning environment and providing substantive feedback to students by acknowledging correct answers in a positive fashion and correcting wrong answers. To assess how well teachers teach, we measure (i) teachers’ pedagogical knowledge, (ii) teachers’ capacity to assess students and monitor their progress, (iii) teachers’ actual use of a monitoring system and knowledge of student performance, and (iv) teachers’ classroom practices based on direct lesson observation. To measure general pedagogical knowledge, we asked teachers to prepare for a lesson with a specified topic by reading and extracting information from a factual text on that topic (general content 72 | The Learning Crisis in Afghanistan knowledge) and to state (in 1-2 sentences) what they would expect their students to learn from the lesson. Both these tasks are consistent with professional tasks normally expected of primary teachers. To measure teachers’ ability to assess students’ learning and give feedback (which we shorten here to “assessing students”), teachers were asked to prepare questions that required students to recall what was learned (lower order) and questions that asked students to apply the material to new contexts (higher order) on the basis of a reading of a factual text. In a second task, teachers were asked to use a marking scheme to give feedback on strengths and weaknesses in students’ writing and to distinguish weak and strong learners. In a third task, teachers were provided with a list of students’ grades; they were then asked to turn the raw scores into averages and to comment on the learning progression of individuals and groups of students with the help of a bar chart. To go beyond capacity and measure whether teachers implement a monitoring system we asked teachers (i) to show us their records of individual students’ performance through the year, and (ii) to estimate their students’ knowledge in three areas of their curriculum. For Language teachers, we asked them the percentage of students that can (a) identify letters, (b) read words and (c) read a paragraph with fluency and answer questions on its content. For Mathematics teachers, we asked them the percentage of students that can (a) add single digits, (b) multiply single digits, (c) multiply double digits. We then compared their answers with their students’ assessment scores on those items. As reported in Figure 2.9, Panel A, teachers struggle both to read and understand a factual text (average score of 40%), and translate this information into teaching; they had a hard time formulating what they wanted children to learn from the lesson based on their own reading of the text (average score of 29% on this task). As with general pedagogical knowledge, the results in Panel B show that few teachers demonstrated an ability to assess student learning and respond to that assessment. A third of teachers could formulate questions that checked basic understanding based on what they had read (35%), and another third could formulate a question that asked students to apply what they had learned to other contexts (29%) — ranging from 23% in co-ed classrooms schools to 42% in girls schools (Table C.9 in the Appendix C). 19% of teachers could give feedback on strengths and weaknesses in students’ writing using a marking scheme. Furthermore, just 2 out of 10 teachers could monitor and comment on the learning progression of students—ranging from 15% in co-ed classrooms schools to 23% in girls schools (Table C.9 in the Appendix C). Panel C shows that teachers do not know what students know or how they have (or have not) progressed over the school year. Only a third of teachers had a record of students’ performance on assessments and homework requested during the year. When asked to identify what their students know or don’t know, no mathematics or language teacher in the sample was able to correctly estimate within a 20-point margin what their students have mastered in all three areas. Around 4 in 10 mathematics teachers correctly estimated the percentage of students that can add single digits, 2 in 10 correctly estimated those that can multiply single digits, and 1 in 10 those that can multiply double digits. The results were even lower in language, where in the three areas (identify a letter, read a word and read a text and answer questions about it), only 1 in 10 teachers could estimate the percentage of their students who could perform one of those tasks. Poor knowledge of general pedagogy was mirrored in teaching behavior in the classroom, as shown in Panel D. Half of the teachers explained the topic of the lesson at the start and summarized what was learned at the end, and around 15% of lessons seemed unplanned to the observers. During their Results of the Afghanistan SABER Service Delivery Survey | 73 lessons, many teachers asked questions that required students to recall information or to practice what was learned, but significantly fewer asked questions that required higher-order skills, encouraged students to apply what was learned to different contexts, or be creative. Overall, 33% of teachers mixed lower- and higher-order questions in their class, ranging from 15% of teachers in co-ed classrooms to 41% of teachers in girls school classrooms (Table C.9 in the Appendix C). Around a quarter of the teachers consistently gave positive feedback and corrected mistakes in response to student answers without scolding students. Figure 2.9: Pedagogical knowledge and skills in the classroom 74 | The Learning Crisis in Afghanistan In summary, general pedagogical knowledge, the ability to assess and respond to students’ learning, the institution of a monitoring system to measure what students know is poor across teachers in Afghanistan. Inside the classroom, many teachers deploy some of the teaching practices identified in the literature as promoting learning, but few (less than two in ten) apply the full set of beneficial skills— structuring, planning, asking questions, creating a positive environment and providing constructive feedback—in their lessons. International Comparison As in the Mathematics content knowledge test, teachers in Afghanistan perform slightly better than the average teacher in Sub Sahara Africa but well below the best performers (Table 2.5). Table 2.5: Comparison of teachers’ performance on pedagogic knowledge and skills in the classroom Afghanistan Average SSA Min Max Panel A: Pedagogical knowledge Factual text comprehension (0-100) 40 47 23 (MOZ) 78 (TZA Formulate aims and learning outcomes (0-100) 29 23 11 (NGA) 41 (TZA) Panel B: Assessing students Formulate questions to check understanding (0-100) 35 23 5 (NGA) 55 (KEN) Formulate questions to apply to other contexts (0-100) 29 7 3 (NGA) 15 (TZA) Assessing students’ abilities 19 19 8 (NGA) 39 (KEN) Evaluating students’ progress 20 12 5 (NGA) 26 (KEN) Panel C: Skills and practices in the classroom Introduce and summarize topic of the lesson (%) 54 41 16 (MOZ) 62 (KEN) Lesson appears planned to enumerator* (%) 85 - - - Ask a mix of lower and higher order questions (%) 33 31 14 (MOZ) 44 (UGA) Positive environment and feedback (%) 24 - - - Engages in all of the above practices (%) 14 9 1 (MOZ) 17 (KEN) Note: The figures from Sub Sahara Africa are taken from Bold et al 2017. For SSA countries there is no data on monitoring systems and teacher knowledge of student performance. Source: SABER Service Delivery - Afghanistan 2017 The Profile on an Afghan Class A subset of 88 Grade 4 classrooms were additionally observed using the Classroom Assessment Scoring System (CLASS). CLASS measures three broad domains of teaching practices: teachers’ emotional support, classroom organization, and instructional support. Each of these is broken down into sub- categories to create 11 constructs on which each classroom is assessed.21 The scores go from 1 to 7, with 7 being the highest. Rather than a checklist tool as the one with which we observed the 170 21 CLASS also has a component on student engagement which we discussed in Chapter 5 of this report. Results of the Afghanistan SABER Service Delivery Survey | 75 classrooms reported above, we videotaped classrooms and then coded them by a certified CLASS coder, which allows us to look at the quality of the teacher-student interactions. Box 2.2 presents what we observed in two average performing classes and one of the relatively higher performing classes for Afghanistan, though still medium performing according to CLASS rubric. We then present the overall scores for our sample of classrooms. Box 2.2: A Tale of Three Classes Subject: Mathematics. Number of Students: 20. In the beginning of the classroom, the teacher arrives and asks individual students to come with their exercise book so they can correct their homework. The teacher takes about 15 seconds per student while the others wait. By the time the teacher is finished, each student has been waiting for about 5 minutes with no learning activity. Then, the teacher takes out the textbook and reads the lesson objective out loud. After that, the teacher asks one student to read the same passage out loud. Then another student, and another, and another until 8 students have read aloud the same passage, spending more than half of the allocated time for the class reading the objective of the lesson. Subject: Language. Number of Students: 10. Upon entering the classroom, the teacher asked questions about the previous class, but these were mostly ineffective at determining student understanding. The teacher then spent a significant amount of time transitioning from this to the new activity. During this time, the teacher silently read the textbook for himself and left students waiting. During the lesson, the teacher asked students to form groups and then midway decided to change the activity and provide new directions for students, thereby confusing some students as to what they needed to do. The classroom was mostly dominated by teacher talk, and we observed no instances of back and forth discussion between teacher and students. Throughout the lesson, the students actively listened, but we did not observe any evidence that the teacher provided students with time to think about the answers before providing the solution. Although the teacher did not successfully manage to maximize instructional time, the Classroom Organization domain is scored at the higher end due to no evidence of misbehavior or negative climate (e.g. misbehavior redirection, punitive control, etc.). Subject: Language. Number of Students: 40. Upon entering the classroom and greeting the students, the teacher asked several close-ended questions about a previous lesson to assess students’ understanding. The teacher then continued by introducing the topic of the new lesson “punctuality” and read the objectives of the lesson from the textbook. The teacher read aloud the entire lesson from the textbook and asked for volunteers to come to the board to also read aloud part of the lesson to their classmates while the teacher monitored and modelled good language skills. The teacher formed groups, provided instructions, and circulated the classroom monitoring their work. The activity consisted of students discussing their daily routine, including what time they get up, what time school starts and ends, etc. After the activity, some students volunteered to present those activities in front of the class. Even though the teacher was leading the discussions, students had opportunities to engage in a back and forth with their teachers and classmates. As in the previous case, there were no instances of misbehavior. 76 | The Learning Crisis in Afghanistan CLASS Overall Results Afghanistan classrooms mostly struggled with low levels of Instructional Support (Figure 2.10). Teachers did not usually make use of a variety of instructional methods, displayed low levels of content understanding, and were not likely to deliver high quality feedback to their students. The area where teachers struggled the most was in providing students with opportunities to think and apply what was learn in a new problem. More than 90% of the teachers received the lowest score on this practice. The level of Emotional Support observed in classrooms was also on the low end. Regard for Student Perspectives, which measures student autonomy and respect and integration of student ideas, was low in over 70% of all classrooms, and the remaining 30% scored on the lower end of the spectrum. Teachers also did not show high levels of sensitivity towards students; the classrooms ranged from low to medium levels. In terms of classroom management, Afghanistan classrooms appeared to be on the higher end of the scale. There were very few instances of negative climate, and teachers were observed to perform well on behavior management. Productivity levels varied with a majority scoring in the mid to upper range of the scale. Results on classroom management should be interpreted cautiously, as an analysis of the reliability of CLASS for the Afghan context showed that while instructional support and emotional support displayed relatively high levels of internal consistency 22 (0.81 and 0.71 respectively), classroom organization showed low levels (0.45) denoting that this domain is not measuring the same latent variable (good teaching) as the others 23. This means that teachers that display good teaching practices, as measured by CLASS, do not score differently on Classroom Organization than teachers that display low teaching skills. The reason is that students do not misbehave in Afghan classrooms regardless of the quality of the teacher. We have observed the same pattern in a study of CLASS carried out in Tanzania primary schools. 22 We computed Cronbach’s Alpha ( = .̅ �+(−1).̅ ) as our internal consistency estimator. His indicator estimates how closely related a set of items are as a group. We do not report inter rater reliability as we only used one coder for all videos to guaranteed consistency in the application of the rubric. 23 The low reliability for Classroom Organization is in line with low correlations between this domain and the others. Results of the Afghanistan SABER Service Delivery Survey | 77 Figure 2.10: Distribution of CLASS domains scores 78 | The Learning Crisis in Afghanistan Below we explain each construct and the results (See also Figure 2.11) Domain I: Emotional Support Dimension Afghan Classrooms Positive Climate: Reflects the emotional connection between While 18% of the teachers do not display affection and rarely the teacher and students and among students. provide positive comments, the rest of the teachers (82%) show some indication of genuine positive affection (e.g. Mashallah), and positive comments (e.g Naam e khoda), but these interactions are brief. Teacher Sensitivity: Encompasses the teacher’s awareness of Around half the teachers observed are sometimes responsive and responsiveness to students’ academic and emotional to students’ needs, although these interactions are not needs. consistent. About 12% of teachers are unresponsive to students’ academic and social needs and fail to notice when students need help. Regard for Student Perspective: Captures the degree to Approximately, 44% of the teachers rigidly provide all the which the teacher’s interactions with students and classroom structure in the class and do not encourage students’ ideas, activities place an emphasis on students’ interests, nor create the opportunities for meaningful peer-peer motivations, and points of view and encourage student interactions. responsibility and autonomy Domain II: Classroom Management Dimension Afghan Classrooms Behavior Management: Encompasses the teacher’s ability to In 90% of the classrooms, no student misbehavior is observed. provide clear behavioral expectations and use effective methods to prevent and redirect misbehavior. Negative Climate: Reflects the overall level of expressed In 98% of the classrooms there is no evidence of negativity or negativity in the classroom. punitive control. Productivity: Considers how well the teacher manages Most teachers are on task and provide students with learning instructional time and routines and provides activities for activities. However, as explained below, the quality of the students so that they have the opportunity to be involved in activities is often low. learning activities. Results of the Afghanistan SABER Service Delivery Survey | 79 Domain III: Instructional Support Dimension Afghan Classrooms Instructional Learning Formats: Focuses on the way in which Teachers fail to use other teaching strategies beyond the the teacher maximizes students’ interest, engagement, and textbook and the blackboard. ability to learn from lessons and activities. Content Understanding: The depth of the lesson content and In nearly 50% of the classrooms, the material and the class the approaches used to help students comprehend the discussion fail to effectively communicate the essential framework, key ideas, and procedures in an academic attributes of concepts/procedures to students, and the focus discipline. is primarily on presenting discrete pieces of information. The broader ideas and/or students’ previous knowledge or potential misconceptions are not presented and/or clarified. For example, in a triple digit subtraction class, the teacher wrote 6 examples on the board, solved one without clearly explaining it, and asked students to solve the remaining 5 examples. Quality of Feedback: Assesses the degree to which the In 75% of classrooms, feedback is non-existent or mechanical. teacher provides feedback that expands learning and In several instances, upon students making mistakes, the understanding and encourages continued participation. teacher either scolded them or asked them to sit down without any attempt to scaffold student learning or help students deepen their understanding. Analysis and Inquiry: Assesses the degree to which students 98% of the teachers present the material without providing are engaged in higher level thinking skills through the students with opportunities to apply those concepts in novel application of knowledge and skills to novel and/or open- open-ended tasks. In the observed classrooms, most of the ended problems. questions asked from students are either closed-ended or require a short yes/no answer. Instructional Dialogue: Content-focused discussion among The class is dominated by teacher talk in 75% of classrooms. teacher and students that is cumulative, with the teacher 25% of the teachers use some facilitation strategy that supporting students to chain ideas together in ways that lead encourages exchanges that build on one another; however, to deeper understanding. these instances are brief and inconsistent. Source: The explanations of the constructs come from the Upper Elementary CLASS manual and is an adaptation from Coflan et al (2018). 80 | The Learning Crisis in Afghanistan Figure 2.11: Distribution of CLASS dimension scores Notes: The figures are computed for a sample of 88 Grade 4 classrooms. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 81 SECTION IV. WHAT EXPLAINS THESE RESULTS? WHY DOES THE SYSTEM USED TO SELECT, TRAIN, AND REMUNERATE TEACHERS NOT PRODUCE HIGH QUALITY TEACHING? Many low-income countries have witnessed a huge expansion in the provision of primary education in the last two decades. In 2016, more than 9.2 million students were enrolled in all levels of school; this represents a 9-fold growth from 13 years earlier. Since 2005, enrollment at the primary level grew every year, sometimes by more than 10% annually. This requires increasing the number of teachers to keep up with increased student enrollment. Such a rapid expansion of the teaching force provides a real opportunity for updating the pipeline—an opportunity that will be lost if the system for selecting, training, and motivating teachers does not ensure there are good teachers in schools. So why does the existing system not produce high quality teaching, as suggested by the evidence presented above? Here, we argue two reasons: the system used to select and induct teachers does not deliver high- quality candidates and the system used to support teachers to deliver high-quality teaching 24 is markedly ineffective. Selection and Induction System De-jure, Afghanistan possess a well-established system of teacher selection and induction. To enter the teaching career, teachers must have completed at least lower secondary education (Grade 12) and participate in a two year program (Grade 14). In our sample, this is true for 85 %of the teachers, while the remainder have only Grade 12 and few teachers have only complete primary school (Grade 6). Figure 2.12: Mismatch between what teachers study and what they teach — Language De facto, however, training systems fall short of international best practice (Bruns and de Luque, 2014). First, standards for entry into teacher training are low compared to high- performing education 24 This section draws on Bold et al (2017). Chapter 3 will discuss teachers’ evaluation and supervision in more detail. 82 | The Learning Crisis in Afghanistan systems around the world. Second, the teacher training programs are often of low quality, delivered by former teachers rather than trained instructors, and ill-suited to the needs of the candidates, who, having gone through their country’s primary and secondary education system, often arrive poorly prepared. They are then confronted by curricula that focus on teaching methods and pedagogy theory rather than content knowledge and practical skills they can use in the classroom (See Box 2.3). In addition, while research suggests that pre-service training that focuses on the work teachers face in classrooms produces more effective teachers and higher learning for students (Boyd et al., 2009), little time is devoted to actual classroom practice. Less than half of the teachers in Afghanistan report having taught a class supervised by an experienced teacher or participating in practicum before starting to teach. Moreover, once in the schools, they are asked to teach something different than their specialization learned during their pre-service training. Only half of the current teachers that teach Language (Mathematics) did their training in Language (Mathematics) (Figure 2.12 and Figure 2.13). Figure 2.13: Mismatch between what teachers study and what they teach – Mathematics In short, it is easy to see how a vicious circle is created in which today’s teachers have gone through an education system that does not adequately prepare them, through a training system with low entry requirements that does not compensate for the flaws in the education system, to be sent into school where they struggle to teach the next generation of students. Moreover, we find that the remuneration scheme is most strongly predicted by experience and age, characteristics which in turn have little systematic relationship with teacher quality. Results of the Afghanistan SABER Service Delivery Survey | 83 Box 2.3: Pre-Service Teacher Training is not associated with better teaching skills Looking at differences in teaching skills by training level, teachers with Grade 12 level education do not have significant differences with those teachers with Grade 14 level training (Figure 2.14). This means that while we observe small differences in teaching practices among these teachers, those differences are driven by sample differences and we cannot say that those two group of teachers are statistically different in terms of them classroom practices. Similar results are found using the CLASS dataset described above. Figure 2.14: Classroom practices by training level Support System To measure teachers’ support system, we focus on four areas: (i) teacher autonomy, (ii) availability of teaching materials, (iii) professional development, and (iv) guidance and support on a day to day basis. To measure teachers’ autonomy, teachers were asked how important their role was in deciding what to teach (curriculum and textbook), how to teach (teaching methods), and who to teach (grade/subject they are assigned to teach). To measure availability of teaching materials, teachers were asked whether they have the teaching material to do their job: curriculum, lesson plan, textbooks that contains classrooms activities, details on what students are expected to learn at each grade (Students’ profiles), guide on how to evaluate students, whether they have a rubric or scoring system to help them evaluate students work, and item banks they could use for quizzes and examinations. To measure access and quality of professional development, teachers were asked if they had participated in professional development during the last school year, how long did it last, what was the content of the training, and the percentage of the training or follow up support that was done in the classroom. Finally, to measure the coaching they received, we asked teachers to reflect in three areas: (a) from all their responsibilities, what is the most important for their principals, (b) whether teachers received 84 | The Learning Crisis in Afghanistan guidance to develop their lesson plan, and (c) the extent to which principals and inspectors evaluate what teachers do in the classroom. As reported in Table 2.6, Panel A, teachers have relatively high levels of autonomy. 70% of the teachers report they have influence on the curriculum and 82% can decide on their teaching methods. More than a third cannot decide which book to use and less than half which grade or subject to teach. Panel B shows that the high levels of autonomy are not complemented by high levels of support. Half of the teachers do not have the curriculum and a third do not have the lesson plan for the day. Half of the teachers do not have textbooks for classroom activities. Furthermore, only few teachers have a document to guide them on how to best assess students, and about 6% of teachers has access to an item bank to facilitate checking for student understanding — ranging from 1 in 10 for teachers in boys schools to no teachers in the sample of girls schools (Table C.10 in the Appendix C). 40% of teachers did not attend professional development during the last school year (Panel C), ranging from 65% for teachers in boys schools to 55% for co-ed classroom schools (Table C.10 in the Appendix C). Those that participated in professional development did so for 12 days on average. However, as in the case of the pre-service teacher training, the focus was on theoretical general pedagogic skills (61%) rather than specific techniques (11%) teachers could use in their next class. A third of these trainings did not involve any practice or follow up in the classroom. Weaknesses in the availability of teaching material and professional development was mirrored in the coaching teachers received, as shown in Panel D. 40% of the teachers did not received any help to develop their lesson plan, which describe what a teacher is going to do in the classroom. Not only this, but 15% of the teachers received no evaluation in the previous school year. This is compounded by the fact that when they are evaluated, roughly half of the time the principal or inspectors/district officers did not evaluate them on what they do in the classroom but on other areas, such as attendance. Finally, providing further evidence that the focus of the schools is not currently on improving teaching and learning, teachers reported that their main responsibility is being on time to school (86%). Only 1% of teachers said that their main responsibility was improve student learning outcomes. In summary, the teachers’ support system is weak across schools in Afghanistan. While teachers have autonomy to choose what and how to teach, they do not receive support in terms of materials, examples of good practices, professional development, or coaching by the school principal or experienced teachers. In light of this evidence, it is no surprise the results we presented in this chapter. Absence, which according to the teachers is their main responsibility, is in fact low. However, improving student learning is not seen as a main responsibility, and teachers and schools are not focused on achieving this goal. Results of the Afghanistan SABER Service Delivery Survey | 85 Table 2.6: Strengths and weakness of the teachers’ support system Indicator Mean Panel A – Teaching Autonomy Curriculum (%) 70.71 Teaching methods (%) 81.56 Textbook (%) 66.21 Grade/Subject to teach (%) 42.38 Panel B – Availability of Teaching Materials Curriculum (observed) (%) 49.66 Lesson plan (observed) (%) 73.38 Guide on how to assess students (observed) (%) 8.18 Student Profiles (%) 30.36 Textbooks for classroom activities (%) 55.71 Rubrics to evaluate for student work (%) 6.18 Item banks for quizzes or examinations (%) 5.27 Panel C – Professional Development Participated in Professional Development (%) 60.14 Average Number of Days 11.88 Main Topic of the Training: General Pedagogical Skills (%) 61 Main Topic of the Training: Content Knowledge (%) 22 Main Topic of the Training: Specific Pedagogical Skills (%) 11 Training or Follow up in the classroom (%) None 29 Less than 24% 7 Between 25%-50% 13 Between 51%-75% 31 Between 76%-100% 19 Panel D – Coaching Most important responsibility: Being on time to school (%) 86 Most important responsibility: Improve students score (%) 1 Guidance to develop their lesson plan (%) 62.74 Evaluated by the principal (%) 86.12 Evaluates by the principal on classroom practices (%) 54.27 Evaluated by the inspector/district officer (%) 85.69 Evaluates by the inspector/district officer on classroom practices (%) 59.01 Source: SABER Service Delivery - Afghanistan 2017 86 | The Learning Crisis in Afghanistan Box 2.4: Notes from the Field A. Professional Development In Kabul city, while supervising the data collection process, I spoke to many of these teachers. Many complained about the quality of these trainings such as: “MoE held a joint training with an international organization for us in three levels. Each level lasted few days and they were a month apart. We were very excited to learn about new methods to improve our skills, but once I attended these trainings the principal trainer kept repeating the same thing for all these levels. I was very disappointed.” Another teacher suggested: “we hope these trainings to be held in school breaks. Many of us cannot afford to be absent from our classes. Some of these trainings are a week long. Our school doesn’t have adequate substitute teachers. I am always worried about my students getting behind while I am in these trainings.” B. Salary and Lesson Plans I spoke to a male teacher in Kabul city all boys school. “I have 5 children, and my salary is 9000 AFG per month. I live in a rented place with my family. I pay 8000 AFG for rent. I need to work extra in other schools as a contract teacher or work at my brother’s repair shop in the afternoons to make ends meet.” When I asked about how much time he dedicates to lesson planning he replied: “very little. Sometimes I read the lesson right before my class or during the class breaks. If my salary was higher and I wasn’t financially struggling, I could focus more on creating lesson plans and new activities for my students.” ____ Contributed by Anahita H Matin Results of the Afghanistan SABER Service Delivery Survey | 87 Chapter 3 : School Management Evidence suggests that effective school management and leadership play a crucial role in school performance. Schools with better management have better learning outcomes (Bloom et al, 2015). The main finding in this chapter is that in Afghanistan, poor school management and lack of accountability are failing students. Despite the fact that Afghan principals work long hours and have low absence rates, there is a considerable mismatch between their professional development, their knowledge of school performance, and their decision-making power. What do Afghan principals do? Most primary school principals (83%) in Afghanistan report having received school management training that focuses in administrative skills, but almost nothing on improving instruction. This means that Afghan principals are not equipped to use their time effectively. The largest proportion of their time, which is a little less than 8 hours, is used to perform school management activities including as managing teachers, school administration, and asset management. However, they also use one and a half hours of their time to teach classes in addition to completing their non-teaching duties as principal. In other words, they are bureaucrats burdened by administrative work and are unable to provide any pedagogical support and coaching to teachers. What do Afghan principals know? Most Afghan principals are not knowledgeable about their school performance in terms of teacher absence, teacher content knowledge and learning outcomes. Only 3% of all principals had correct knowledge of the school’s performance in all these 3 areas. In particular, almost all Afghan principals think that the majority of Grade 4 students in their school would know how to solve a single digit addition, but only 41% of principals are actually correct about this statement. Similarly, almost all Results of the Afghanistan SABER Service Delivery Survey | 89 Afghan principals think that the majority of their teachers would know the answer to a double digit subtraction, but only 35% of principals are knowledgeable about teacher’s performance in this task. This poor knowledge of teacher’s individual abilities and overall learning outcomes is suggestive evidence that can also be closely linked to the lack of adequate training; very few Afghan principals follow good school management practices. For instance, even though many principals are using classroom observation, they do not have the proper training to correctly implement it since their training is mainly focused on administrative and HR management skills. Only 13% of Afghan principals qualified as following good practices of teacher evaluation. Almost all principals in Afghanistan reported that teachers’ most important responsibility is being on time to school (88%), followed by maintaining strict discipline in the classroom (7%), and lastly, teaching students to be good citizens (5%). Surprisingly, none of the principals considered that improving students’ learning should be the main focus of teachers’ duties. How much decision-making power do Afghan principals have? Principals in Afghanistan also lack considerable autonomy and decision-making power as most of the important decisions associated with student learning goals are made by the Ministry of Education. Principals report having some autonomy mainly in administrative tasks. In this aspect, there is an apparent mismatch between the topics on which principals are trained and the areas they feel they have a say. For instance, Afghan principals report having a lot of decision-making power when appointing a teacher to participate in training, but less than half of the principals have taken the necessary trainings that would allow them to make a good decision about which training teachers should do. Are Afghan principals accountable? Even though Afghan principals report high collaboration with the community through the school shura and the teacher-parent associations, it is less apparent that these actors have much authority to hold schools accountable. In Afghanistan, the school management committee and the community are the lowest decision-making actors. This chapter is organized as follows. The first section provides information on principals’ professional development and effective use of time. The second section focuses on principals’ ability/behavior in terms of their knowledge of school performance and whether they are following good management practices. The third section analyzes principals’ autonomy and the relative decision-making power of different actors in the education system. Lastly, the fourth chapter discusses accountability mechanisms in terms of administrative, pedagogical, and content knowledge supervision visits from local education entities. Information about parental and community participation with schools is also discussed in this chapter. 90 | The Learning Crisis in Afghanistan SECTION I. WHAT DO PRINCIPALS DO? UNDERSTANDING PRINCIPALS’ PROFESSIONAL DEVELOPMENT AND EFFECTIVE USE OF TIME Approximately, 83% of principals have received some kind of school management training, which in general is quite short. (Figure 3.1). The majority of this type of training (88%) lasts about 1 to 4 weeks long. Only 12% of principals have taken school management training that lasts more than 1 month. Figure 3.1: Primary school principals receiving management training and average duration Formal Training in the DuraƟon of the Training Management of a School 17% 83% 12% 88% YES MORE THAN 1 MONTH NO 1–4 WEEKS 100% 94.2*** 83 80% PERCENTAGE OF PRINCIPALS 71 60% 40% 20% 0% All Urban Rural Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 91 The largest difference in school management training is found between rural and urban principals. Being a rural principal decreases the likelihood of receiving school management training by 23 percentage points, on average, and this result is statistically significant at the 1% level. Most primary school principals in Afghanistan report having received school management training that focuses in organizational leadership, but almost none in instructional leadership. Figure 3.2 shows the main type of professional development training taken by primary school principals in Afghanistan. Most Afghan principals have taken training in administrative skills followed by some training in HR management skills (39.1%). In contrast, training in financial skills and communication skills is minimal. These results indicate that Afghan principals do not receive any training in supporting teachers to improve their teaching skills. Figure 3.2: Principals’ professional development Source: SABER Service Delivery - Afghanistan 2017 Principals in Afghanistan work long hours. Figure 3.3 shows the distribution of hours worked per day for primary school principals. In Afghanistan, around 60% of principals reported spending over eight hours per workday on school related tasks. In particular, principals in Afghanistan work on average 8.63 hours, which is normally done on a full-time basis. Some principals may work as much as 12 to 15 hours. In general, principals in urban schools work at least one additional hour compared to those in rural schools. 92 | The Learning Crisis in Afghanistan Figure 3.3: Hours per day principals spend in school and use of time Source: SABER Service Delivery - Afghanistan 2017 Principals in Afghanistan often juggle several tasks on a daily basis. The largest proportion of their time (83%), which is a little less than 8 hours, is used to perform school management activities including managing teachers, school administration, and asset management. However, they use 17% of their time to teach in addition to completing their duties as principal. This is equivalent to around one and a half hours of teaching per day. There is no statistical difference in teaching hours between urban and rural principals. However, principals in urban areas tend to spend one and half hours more in non-teaching activities compared to principals in rural areas. Moreover, there are no distinguishable differences in principals’ use of time by type of school in terms of gender mix (Table 3.1). Table 3.1: Principals’ use of time per day by urban/rural Diff. R/U All Rural Urban Mean Principals’ Total Hours Worked 8.63 8.01 9.45 (1.44)*** Principals’ Teaching Hours 1.55 1.90 1.58 (0.32) Principals’ Non-Teaching Hours 7.70 6.12 8.27 (2.14)*** Source: SABER Service Delivery - Afghanistan 2017 Principal and Head Teacher Absence Rate There are three main primary school leadership positions in Afghanistan: (i) principal, (ii) deputy principal and (iii) head teacher. 86% of the public schools are led by principals, 1% are led by deputy principals and 14% are led by head teachers. The positions of deputy principal and head teachers are more likely to be present in rural areas (Table 3.2). Results of the Afghanistan SABER Service Delivery Survey | 93 Table 3.2: Number of primary school leadership positions by urban/rural Leadership Position Urban Rural Total % Principal 71 75 146 86% Deputy Principal 0 2 2 1% Head teacher 3 19 22 14% Total 74 96 170 Source: SABER Service Delivery - Afghanistan 2017 Principal absence rate in Afghanistan is relatively low (6%). Principal absence rate is measured as the share of school principals (principal and deputy principal) who are absent from school at a time of an unannounced visit (second visit). Principals who were "at school but not his/her shift" are classified as not absent from school. Figure 3.4 presents principal and head teacher absence rate by urban/rural schools. In Afghanistan, the principal absence rate is observed to be around 6%, with rural schools having slightly higher principal absence rate than urban schools. However, this difference is not statistically significant. Figure 3.4: Principal and head teacher absence rate Note: Principals and Head Teachers who are “at school but not in their shift” are classified as not absent from school. Source: SABER Service Delivery - Afghanistan 2017 Head teacher absence rate is significantly higher compared to principal absence rate. In this case, we also define head teachers who are at school but not in their shift as non-absent from school. In Afghanistan, 22% of head teachers are absent from school at a time of an unannounced visit. Head teacher absence rate is significantly higher in urban schools (28%) compared to rural schools (15%). This is likely due to the fact that some rural schools have only a head teacher as school leader instead of a principal, who performs the duties of principal. While urban schools in general not only have a principal but also a head teacher. 94 | The Learning Crisis in Afghanistan Box 3.1: Profile of Afghanistan’s Principals In Afghanistan, principals are significantly more likely to be male, especially in rural schools. Out of the total 170 primary school headmasters (including principals, deputy principals and head teachers) that were surveyed in the Afghanistan SABER SD 2017, only 25% were female, suggesting a strong male dominance among school principals in this country. This male dominance is even more pronounced in rural schools, where almost 90% of school principals are male. In urban schools, it is more likely to find female principals (39%), but the difference with rural schools is not statistically significant (Figure 3.5). Figure 3.5: Percentage of male and female principals/deputy principals/head teachers Source: SABER Service Delivery - Afghanistan 2017 Figure 3.6 shows principals’ level of education in Afghanistan. The majority of principals (approximately 90%) fulfill the minimum education standard for becoming a teacher (Grade 14). Only 3% of primary school principals have an advanced diploma in teaching or a higher degree (i.e. master’s degree). Figure 3.6: Principals’ level of education in Afghanistan Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 95 In Afghanistan, there is no separate or specific principal career track in order to become a principal. In other words, principals are typically promoted from the pool of senior teachers, which means that the principal workforce tends to be older, and not far from retirement. Figure 3.7 shows what position principals held in their previous assignment or in a teaching position. Clearly, most principals in Afghanistan (78%) were in a senior teacher position before becoming a principal. Figure 3.7: Positions before becoming primary school principal Source: SABER Service Delivery - Afghanistan 2017 Figure 3.8 shows the distribution of principals’ ages in Afghanistan. The average age of principals in Afghanistan tends to be at 45-50 years. About 37% of principals are above 50. Figure 3.8: Distribution of principal’s age in Afghanistan Source: SABER Service Delivery - Afghanistan 2017 96 | The Learning Crisis in Afghanistan A teacher in Afghanistan needs on average 12 years of experience in the education sector to acquire the position of principal. However, a rural teacher may become principal much sooner than an urban teacher. In particular, rural teachers need only 9.93 years of experience, while urban teachers need on average 14 years of experience. This number can be even lower for principals of Co-educational schools (in the classroom), where a teacher can become a principal in only 5-6 years. This difference in years of experience also applies to gender. Female teachers require around 2 more years of experience than male teachers before becoming principals (Figure 3.9). Figure 3.9: Distribution of years of experience before becoming a principal by gender and rural/urban Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 97 Principals in Afghanistan are paid relatively well compared to teachers. They earn almost 2 times more than teachers, especially in urban areas. Specifically, the average principal salary is about 183 USD, with a dispersion from 58 USD to 570 USD (including allowances). Urban principals earn significantly more than rural principals. Half of Afghan principals reported to have received their salary with delays during the last year (Figure 3.10). Figure 3.10: Comparison of Principal’s and Teacher’s Monthly Salary in Afghanistan Source: SABER Service Delivery - Afghanistan 2017 SECTION II. WHAT DO PRINCIPALS KNOW? PRINCIPALS’ ABILITY IN SCHOOL MANAGEMENT In order to measure principals’ ability and skills in school management we use two types of indicators: (1) principal’s knowledge of school performance, and (2) principal’s management practices. Principals’ Knowledge of School Performance The first indicator assesses principals’ knowledge of school performance in three critical areas: (i) teacher presence, (ii) teacher content knowledge, and (iii) student and teacher learning outcomes. To compute this index, we combine information gathered from a school principal survey with actual data on student and teacher performance. (i) Teacher Absence: This indicator is measured using the average absolute difference between principals’ assessment on teacher absence and the actual teacher absence rate on the day of the unannounced visit. In order to measure principals’ knowledge about teacher absence, we use the following question asked to the school principal: Out of all the teachers that work in this school, usually what percentage of them is absent from the school at any given day? The 98 | The Learning Crisis in Afghanistan grading criteria for the principal’s answers is the following: if the principal’s answer falls on or within 5 percentage points (+/-) of the actual teacher absence rate on the day of the unannounced visit, he/she is scored as knowledgeable (1), otherwise he/she is scored as not knowledgeable (0). (ii) Teacher Ability – Content Knowledge: This indicator is measured using the average absolute difference between principal assessment of teachers’ content knowledge and the information coming from the test administered to the teachers. The principal is shown the teacher assessment in the subject content area that the teacher masters (Mathematics, Language or both), and then he/she is asked to answer the following question for several of the teachers in his school: As part of this survey we will assess teacher knowledge in Mathematics and Language. For each of the following teachers, how do you think each of them will score in the assessment? The grading criteria for principal’s answers is the following: on or within (+/-) 10 points of the category chosen by the principal is scored as knowledgeable (1), otherwise is scored as not knowledgeable (0). If the principal is knowledgeable of at least 60% of the teachers for whom he/she has provided an opinion, he is scored as knowledgeable. (iii) Learning Outcomes (Teachers and Students): This indicator is measured using the average absolute difference between the principal assessment of student and teacher skills and the actual information coming from the test administered to the students and teachers. For this question the principal is asked the following questions on teacher and student learning outcomes, respectively: (a) If I asked the Mathematics teachers in this school to subtract double digit numbers (e.g. 57-49=?), what percentage of them would get the right answer? and (b) If I asked Grade 4 students in this school to add single digit numbers (e.g. 8+7=?), what percentage of them would get the right answer? The grading criteria for principal’s answers is the following: on or within 10 percentage points (+/-) for each question is considered to be knowledgeable (1), otherwise he/she scored as not knowledgeable (0). Finally, a principal is coded as knowledgeable if he is knowledgeable in all three areas. All three areas carry the same weight in the aggregation process. Principal knowledge of school performance is very low in Afghanistan. No principal had a correct knowledge of the school’s performance in all these 3 areas: (i) teacher presence, (ii) teacher content knowledge, and (iii) student and teacher learning outcomes. Figure 3.11 shows principal knowledge of school performance in Language and Mathematics. First, the results indicate that less than half of all school principals (45%) were knowledgeable about the rate of teacher absence in their school. Second, around half of the school principals were knowledgeable about their mathematics teachers’ individual abilities (53%), and they were knowledgeable about their language teachers’ individual abilities only 8% of the time. Lastly, in terms of learning outcomes, 42% of all principals were knowledgeable about their students’ performance in single digit addition, and around 35% were knowledgeable about their teachers’ knowledge of double digit subtraction. Results of the Afghanistan SABER Service Delivery Survey | 99 Figure 3.11: Principal knowledge of school performance in Language and Mathematics Note: Principals and Head Teachers who are “at school but not in their shift” are classified as not absent from school. Source: SABER Service Delivery - Afghanistan 2017 Principal Knowledge of Learning Outcomes (Students and Teachers) Since learning is the most important component of this indicator, we compare principals’ predictions of student and teacher skills to the actual performance of students and teachers at the school level disaggregated by performance score interval. Figure 3.12 represents principal knowledge of student performance in single digit addition, Grade 4 students’ actual performance on the test at the school level, and principals’ prediction of student performance for each score interval. And, Figure 3.13 represents principal knowledge of teacher performance in double digit subtraction, teachers’ actual performance on the test at the school level, and principals’ prediction of teacher performance for each score interval. In general, almost all Afghan principals think that the majority of Grade 4 students (i.e. 50% to 100%) in their school know how to solve a single digit addition. Even though in more than 90% of the schools, more than half of Grade 4 students are able to solve the single digit addition, only 41% of principals were able to accurately estimate the percentage of their students who could perform the task within 10 percentage points. 100 | The Learning Crisis in Afghanistan Most Afghan principals tend to significantly overestimate student performance, especially for those in the top 1% of the test score distribution. For instance, in almost one-fifth of the schools (19%), principals predicted that all Grade 4 students in their school would know the answer to a single digit addition. However, the results from the survey indicate that in only 3% of the schools, all Grade 4 students were able to solve this task and in only 4% of the schools, principals were actually correct in their estimation. In addition, almost all Afghan principals estimated that more than half of their Grade 4 students would master the single digit addition exercise. The results of the survey suggest that in 92% of the schools, most of Grade 4 students were able to solve the problem, but once more, only 37% of the principals were able to predict correctly. Lastly, in only 2% of schools, principals predicted that less than half of their Grade 4 students could not master this basic Mathematics exercise. The student assessment results indicate that in 8% of the schools less than half of the students could not perform correctly. For the bottom 50% of the test score distribution, principals were not able to accurately predict any of the results. Figure 3.12: Principal knowledge about students’ performance in single digit addition Note: The principal answered the following question: “If I asked Grade 4 students in this school to add single digit numbers (e.g. 8+7=?), what percentage of them would get the right answer?”. The Y-axis represents percentage of schools, while the X-axis represents the interval score for three different variables: percentage of students in each score interval, percentage of principals predicting student’s performance and percentage of principals predicting student’s performance correctly. Source: SABER Service Delivery - Afghanistan 2017 Similarly, almost all Afghan principals think that the majority of their teachers (i.e. 50% to 100%) would know the answer to a double digit subtraction. Even though in 95% of the schools, more than half of the teachers are able to solve the double digit subtraction, only 35% of principals are knowledgeable about teacher performance in this task. Afghan principals also tend to overestimate teacher performance in double digit subtraction, especially in the top 10% of the teacher test score distribution. In particular, in 42% of the schools, principals predicted that all their teachers would master the double digit subtraction. According to the teacher assessment, in 70% of the schools, all teachers were able to solve the double digit subtraction. Results of the Afghanistan SABER Service Delivery Survey | 101 However, in only 30% of the schools, principals were correct in their estimation. Similarly, in almost all schools, principals thought that at least half of their teachers (i.e. 50%–99%) were able to solve this problem. The data from the teacher assessment suggests that in 95% of the schools, at least half of the teachers could perform the task. However, only in 34% of the schools, principals were accurate in their estimation. Lastly, in just 1% of the schools, principals consider that less than half of their teachers will not be able to solve the double subtraction problem. The results from the survey indicate that in 5% of the schools, less than half of the teachers are not able to perform this task. For the bottom 50% of the teacher test score distribution, principals seem to be a bit less knowledgeable about teacher performance, as just 1% of principals were able to predict this number correctly. Figure 3.13: Principal knowledge about teacher knowledge in double digit subtraction Note: The principal answered the following question: “If I asked the Mathematics teachers in this school to subtract double digit numbers (e.g. 57-49=?), what percentage of them would get the right answer?” The Y-axis represents percentage of schools, while the X-axis represents the interval score for three different variables: percentage of teachers in each score interval, percentage of principals predicting teacher’s performance and percentage of principals predicting teacher’s performance correctly. Source: SABER Service Delivery - Afghanistan 2017 102 | The Learning Crisis in Afghanistan Do Afghan Principals Follow Good Management Practices? The second indicator of principals’ ability in school management is designed to measure whether the principal follows good management practices. This is a binary index which takes the value of 1 if the school principal does the following three practices, and whether teachers’ actual responses corroborate the principal’s responses: (i) Practice 1: The principal has met with at least half of the teachers during the last school year to evaluate their performance and at least half the teachers say they were evaluated in the past year. (ii) Practice 2: The principal uses at least two of the following methods to assess teacher performance — (a) classroom observation, (b) teacher performance, (c) student assessment and the teacher mentions that these evaluation methods were used as well. (iii) Practice 3: The principal conducts classroom observations regularly (at least once a month per teacher) and provides written feedback and at least half of all teachers mention classroom observation as an evaluation method. Figure 3.14: Principals practices of teacher evaluation Notes: Principal answers checked against teachers answers to similar questions. Source: SABER Service Delivery - Afghanistan 2017 Only 13% of Afghan principals qualified as following good practices of teacher evaluation (Figure 3.14). Most of the Afghan principals follow two of the good practices. More specifically, three- quarters of all school principals meet with at least half of all teachers to evaluate their performance and around 85% of them conduct regular classroom observations for their teachers. Even though Afghan principals seem to follow two of the good practices, the problem is how they are implementing them. In other words, the main issue is that even though many principals are using classroom Results of the Afghanistan SABER Service Delivery Survey | 103 observation, they do not have the proper training to implement it correctly. This is particularly problematic, as Strong et al.’s (2011) experimental research shows, principals who are not trained in classroom observations do not know what to look for during an observation. As a result, rather than supporting teachers, they may end up providing counterproductive feedback. In the study, principals were shown videos of teachers with ranging levels of value-added, and were asked to categorize them “effective” or “ineffective” accordingly. The study revealed principals were below the 50% threshold in correctly identifying effective and ineffective teachers, which is akin to pure chance. In addition, Afghan principals tend not to use varying methods to assess teacher performance. The least number of principals follow the practice of using a variety of evaluation methods to assess teacher performance. On average, 83% of teachers use less than 2 methods of evaluating teachers. As a result, only 13% of all principals qualify as following good practices of teacher evaluation. In Afghanistan, SABER SD 2017 data suggest that while several principals follow at least one of three good management practices, they nevertheless have a poor grasp on overall teacher performance. To further corroborate the fact that Afghan principal management practices are not improving teaching and learning, we use data on principals’ ranking of teachers’ responsibilities. For this purpose, the principal is asked to rank the following 6 teacher activities from most important to least important: (a) being on time to school (b) maintaining a strict discipline in the classroom (c) teaching students to be good citizens (d) improving students’ scores (e) having positive relationships with other teachers (f) helping out in extra-academic activities that the school faces (Figure 3.15). Similar to the teachers’ response in Chapter 2, almost all principals in Afghanistan reported that teachers’ most important responsibility is being on time to school (88%), followed by maintaining strict discipline in the classroom (7%) and lastly teaching students to be good citizens (5%). Surprisingly, none of the principals considered that improving students’ learning should be the main focus of teachers’ duties (Figure 3.15). All Afghan principals ranked “improve students’ scores” in second place or lower (7% in second place, 26% in third place, 22% in fourth place, 21% in fifth place and 22% in sixth place). These results are a clear indication that what matters the most for both, principals and teachers in Afghanistan, is not what is done in the classroom but rather other activities such as attendance. These results could be linked to the low principal and teacher absence rate in Afghanistan. 104 | The Learning Crisis in Afghanistan Figure 3.15: Principals’ opinion on teachers’ most important responsibility Source: SABER Service Delivery - Afghanistan 2017 SECTION III. PRINCIPALS’ DECISION-MAKING POWER In order to measure principals’ decision-making power, we analyze whether he/she has any control, power, or influence when deciding over the following items: (1) school material and infrastructure like the purchase of school books, maintenance of school buildings, etc., (2) teacher management issues like promotions, transfers, etc., (3) teacher contracting issues such as hiring and firing, setting salary, etc., (4) budget decisions, and (5) curriculum development. For this purpose, we use two different types of decision-making measures: (i) principals’ self-reported outcome of decision-making power and (ii) a principals’ self-reported outcome of participation in decision-making. For the first measure, the principal is asked who has the power to decide the above school management items among a set of different actors that include Ministry of Education (central level), provincial level, district level, principal level, school management level, parent association (community), teachers, NGOs, private sector, and international organizations. We define that a principal has decision making power each time the decision is taken at the principal level. For the second measure, the principal is asked how important is his voice in cases he considers that he does not have any decision-making power. In other words, even though the principal might consider that he does not have decision-making power, we measure whether his voice matters a lot in the decision- making process. Therefore, we define that the principal has a certain autonomy in the decision-making process when he/she has the decision-making power over a school management item combined with the cases when his/her voice matters a lot despite not having any power. Results of the Afghanistan SABER Service Delivery Survey | 105 Figure 3.16: Principal decision-making power Source: SABER Service Delivery - Afghanistan 2017 106 | The Learning Crisis in Afghanistan Figure 3.16 shows principal decision-making power for a number of school management items using two different types of measures. The first one (blue bars) represents the percentage of principals who consider having decision-making power, while the second one (red bars) represents not only the percentage of principals who consider having decision-making power but also those who consider that their voice matters a lot in the decision process despite not having any power. Principals in Afghanistan report having some autonomy mainly in organizational/administrative tasks. However, this decision-making power is not associated with factors that directly affect student learning at the school. Afghan principals surveyed in the 2017 Afghanistan SABER SD felt they had considerably more control over administrative issues such as school maintenance, teacher allocation to grades, hiring of non-teaching staff, and developing course content for non-core curricula versus those pertaining to teacher management, such as hiring and retaining effective teachers or dismissing ineffective ones. For instance, over 65% of principals reported looking after the maintenance of the school, versus only 16% for issues related to teacher hiring/dismissal. Most of Afghan principals report feeling constrained in managing poorly-performing teachers. In particular, less than 2% of principals reported complaining about a poorly-performing teacher to a higher official, about 6% considered issuing a warning, while 82% reported speaking to the teacher as the first course of action. Moreover, most of Afghan principals (88%) have never dismissed a teacher during their time as principal of the school. Even when we consider the cases when their voice matters a lot, the only instructional task that Afghan principals are engaged in is the appointment of a teacher to participate in training. When Afghan principals decide to send a teacher for training, their main reasons are to develop teaching methods and lesson plans (Figure 3.17). Despite this, it is hard to believe that this teacher training does not contribute to students’ learning outcomes due to the fact that Afghan principals have a very little knowledge of school performance, especially teacher performance. Results of the Afghanistan SABER Service Delivery Survey | 107 Figure 3.17: Principal’s main reasons for sending a teacher for training Source: SABER Service Delivery - Afghanistan 2017 In Afghanistan, there is a mismatch between the topics on which principals are trained and the areas where they feel they have a say. While most principals in Afghanistan report having received training (83%), this training does not prepare them adequately for the tasks they consider having decision- making power. For instance, Afghan principals report having a lot of decision-making power when appointing a teacher to participate in training, but less than half of the principals have taken a training in HR management skills and communication skills. In addition, some principals also train in financial skills but in reality, their decision-making power on school budget issues is minimal. Distribution of Decision-Making Power in School Management The Ministry of Education in Afghanistan is twice as likely to be the decision-maker on most of the school management issues than principals. The decision-making power of the School Management Committee and the community are minimal at best. Figure 3.18 shows the distribution of the decision-making power on school management issues among 4 main actors of the education system: Ministry of Education, the principal, the School Management Committee, and the community. Clearly, the governance of the schools is largely centralized around the Ministry of Education, which oversees a large portion of the school management areas, including the purchase of books, school supplies and equipment, teacher allocation to schools, teacher promotion and transfers, teacher salary, teacher hiring/firing, and most importantly, preparing and approving school operating budget. 108 | The Learning Crisis in Afghanistan Figure 3.18: Distribution of decision making power in school management areas Source: SABER Service Delivery - Afghanistan 2017 We also analyze whether Afghan principals have dealt with any of the school management issues for which they report having decision-making power or a lot of voice. Afghan principals seem to have even less decision-making power than what they think they have. Figure 3.19 compares the school management issues for which the Afghan principals report having the decision-making power and lot of voice with the actual decision-making cases the principals have dealt with. In general, the school management issues that they have dealt with coincide with the ones for which they report having decision-making power or for which their voice matters a lot. However, there are also some cases for which they report having some decision power, but they have never dealt with this type of issue or the cases of decision-making are very few. For instance, around 20% of Afghan principals report having a lot of voice in the preparation, approval, and implementation of the school budget, but none of them has ever dealt with this type of case. Similarly, around 30-40% of principals report having decision-making power over purchase of school supplies and equipment, teacher promotion, or hiring non-teaching staff; however, only half of them have actually had to deal with Results of the Afghanistan SABER Service Delivery Survey | 109 these types of school management cases. Also, 24% of principals say that their voice matters a lot when purchasing school books, but only 1% has really treated this case. Figure 3.19: Actual decision-making cases for the Afghan principals Source: SABER Service Delivery - Afghanistan 2017 110 | The Learning Crisis in Afghanistan Box 3.2: Principals' letters to The World Bank One school principal for a girls only school in Kabul city told our team: “Lack of proper infrastructure and resources in schools will not allow us to provide students with an environment necessary to improve their learning experience. I personally have written several letters to the Ministry of Education to ask for financial support, but I never got any answer back. My power to manage the budget is limited. It is not me who determines the school budget; it is the Ministry of Education. If the school had enough budget, I could purchase benches and desks for all students.” During the fieldwork, we received several letters from school principals across different provinces of Afghanistan. A few samples of these letters can be seen in the picture above. Our team has read all the letters and found the following areas addressed repeatedly as everyday challenges by school leadership: 1. Lack of proper school building (for schools far from the Capital, many principals mentioned lack of adequate UNICEF tents to protect Principals’ letters to the World Bank students from dirt, wind, dust, rain, etc.) 2. Shortage of desks and textbooks (one school principal wrote to us: “More than 50% of the students have no proper seats. Only students in Grade 8 and 9 get to sit on desks. Two months have passed from the beginning of the school year and we are still waiting for textbooks for some of the subjects.”) 3. Lack of enough Tashkeel positions (Most principals wrote about this problem; some mentioned they need deputy principals, and many principals emphasized they need more teachers for specific subjects such as math, science, chemistry, biology and physics. 4. Lack of female teachers in their workforce/school (In Afghanistan, girls classes are taught by both male and female teachers. Since, there are shortages of teachers in certain districts and provinces, most girls classrooms are taught by male teachers. This has created problems, especially in girls only schools, as some families have prevented their daughters from continuing their education. The social and cultural norms mandate girls education is provided by female educators. 5. Lack of proper management and resource allocation at district and provincial level. 6. Lack of necessary school inputs such as: electricity for computers, border walls to protect students, sanitary latrines and water. (One principal mentioned: “The schools’ water well dries up during summer and the students are left thirsty.”) Results of the Afghanistan SABER Service Delivery Survey | 111 In the Afghanistan SABER SD 2017, primary school principals report several constraints to the proper functioning of their schools. Indeed, the top three constraints are closely related to the principals’ letters and in order of importance they are the following: (1) availability of teacher material, (2) number of teachers and (3) teacher knowledge/pedagogy. Figure 3.20: Principal’s main constraints to improve student’s learning Source: SABER Service Delivery - Afghanistan 2017 ____ Contributed by Anahita H Matin and Field Team 112 | The Learning Crisis in Afghanistan SECTION IV. ACCOUNTABILITY: DISTRICT SUPERVISION VISITS AND COMMUNITY PARTICIPATION On average, almost all schools (98%) in Afghanistan received at least one type of supervision visit — pedagogical, administrative or content knowledge. 25 During the year of the survey, 20% of schools received a pedagogical visit, 56% received an administrative visit and 22% of the schools received a content knowledge visit. In general, schools receive around 4-5 supervision visits during an academic year. For some cases, they can range from 2 to 9 supervision visits per year (Figure 3.21). Table 3.3: Type of supervision visits as reported by the principal Pedagogical Administrative Content knowledge % Received supervision visit (last) 20% 56% 22% Number of visits per year 4.7 4.0 5.6 % Received recommendations 90% 87% 96% % Shared recommendations with staff 77% 85% 98% Source: SABER Service Delivery - Afghanistan 2017 Figure 3.21: Frequency of supervision visits per school year Source: SABER Service Delivery - Afghanistan 2017 From all the recommendations received from the District Education Office through supervision visits, there is a significantly smaller percentage of recommendations that were actually observed by the school staff in written feedback. Around 90% of the principals report that these supervision visits result in recommendations, but only 60% of them consisted of written feedback. The majority of the school principals also report that recommendations are shared with the staff at the school. However, 25 Only 3 public schools from the 170 public schools surveyed in the Afghanistan SABER SD did not have any type of supervision visit. Results of the Afghanistan SABER Service Delivery Survey | 113 after asking a more detailed question of whether teachers receive the written report, the numbers drop to around 30%. Principals in Afghanistan report not only to the District Education Office but also to school shuras. The school shuras, made up of parents and community elders, are community-based school management committees. They are engaged in local campaigns to enroll students, provide school protection, help with school construction and maintenance, and monitor education quality and delivery. Figure 3.22 shows the community participation in Afghanistan in terms of school management committee and parent teacher association. Nearly 90% of public schools in Afghanistan report having a school management committee or shura. Urban schools are slightly more likely to have a school management committee, but there is no statistical difference when compared to rural schools. Similarly, 88% of the schools have a parent teacher association. Urban schools are significantly more likely to have a parent-teacher association compared to the rural schools. Figure 3.22: Community involvement Source: SABER Service Delivery - Afghanistan 2017 Afghan principals report high compliance with community involvement through the school shuras and the teacher-parent associations. However, it is less apparent that these actors have much authority to hold schools accountable. Figure 3.23 shows the main areas of discussion that the school principal has with these two actors. Basically, the SMC and the PTA are concerned about student performance and student attendance. According to data gathered in the SABER SD, most principals report that they have a follow-up structure put in place in order to discuss the recommendations made at the school. 114 | The Learning Crisis in Afghanistan Figure 3.23: Main areas of discussion with the SMC and PTA Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 115 Chapter 4 : Schools Inputs Afghanistan has made significant strides in improving primary education enrollment in the past 15 years. In 2001, one million Afghan students, almost none of them girls, were enrolled in 3,400 schools. In 2015, there has been a nine-fold increase in enrollment with more than eight million students in 16,400 schools, of whom almost 40% are girls.26 Even though, it is encouraging to see progress in access to education, this expansion has not kept pace with school inputs availability in Afghanistan. The main finding in this chapter is that public schools in Afghanistan lack the basic necessary infrastructure to provide students with an adequate learning environment in the school and in the classroom. What school inputs are available in Afghan schools? Despite the surge in reconstruction of schools in Afghanistan through programs such as EQUIP27, only one-third (35%) of public schools have minimum infrastructure availability, which captures the availability of functioning toilets and classroom visibility. Schools in Afghanistan are not equipped with sufficient number of classrooms. Less than three-quarters of the schools in the sample (70%) have permanent classrooms (i.e. complete, with floors and walls), while 15% have semi-permanent classrooms (i.e. incomplete, plastered without shutters or floor, etc.) and 15% of the schools have temporary classrooms. As mentioned in Chapter 1, the ratio of students per class is close to the norm 26 Education Quality Reform in Afghanistan (EQRA) Project Appraisal Document (World Bank, 2017) 27 Education Quality Improvement Program (EQUIP) is a World Bank project that aims to increase equitable access to quality basic education for students in Afghanistan. EQUIP is implemented by the Afghanistan Ministry of Education and funded by the Afghanistan Trust Fund (ARTF). EQUIP supports building new school infrastructures as well as improving school facilities (e.g. building library, extra classrooms, laboratory, computer purchase, etc.) and teaching materials. Results of the Afghanistan SABER Service Delivery Survey | 117 of 1:40, but in urban schools, classrooms can be very overcrowded. Furthermore, the availability of toilets, drinking water and electricity in Afghanistan is very troubling. Overall, almost half (46%) of Afghan schools do not have at least one functioning toilet. Even in schools with such facilities, there are around 169 students per every toilet. This number can be even higher in urban schools, and especially girls schools, where there can be as many as 231 students per toilet. In addition, around 70% of public schools lack clean drinking water and working electricity. In Afghanistan, 80% of the schools have boundary walls and 77% have a security guard at the door. Urban schools are better protected compared to rural schools. Despite Afghanistan’s fragility, only 10% of public schools have a safe shelter in order to protect student’s lives. Lastly, on a national scale, 40% of public schools own a computer, but only 5% of these computers are connected to internet. Urban schools and boys schools are generally better equipped with computers compared to other types of schools. What classroom inputs are available in Afghan schools? Classrooms in Afghanistan also lack the necessary inputs to provide students with an appropriate learning environment. Overall, 35% of the students do not have proper seats and desks. On the positive side, around 89% of the classrooms are equipped with a functioning blackboard and almost all schools have chalk and/or marker available in the classrooms. Around 80% of classrooms in Afghanistan are sufficiently visible to the students, meaning that it was possible for students to read a printout placed on the blackboard from the front as well as from the back of the classroom. Corner libraries in classrooms is practically non-existent in Afghanistan, where only 1% of urban classrooms are equipped with such facilities. On average, one-third of Afghan classrooms displayed some kind of educational material on the walls, such as artwork, charts, maps, etc. In terms of classroom hygiene, 8% of the classrooms were extremely clean and well maintained, 80% were reasonably clean and 13% were not very clean or maintained. Half of Grade 4 Afghan students wear uniforms to school. Afghanistan seems to have made some progress in terms of having access to inputs for teaching, as approximately 86% of Grade 4 Afghan students have textbooks available in the classroom and almost three-quarters of teachers had their lesson plan ready and available. This chapter is organized as follows. The first section describes the availability of inputs for learning at school level. The second section provides information on the availability of learning inputs at the classroom level, including schools inputs for teaching. Finally, the third section compares Afghanistan’s school inputs to other countries surveyed by the SDI survey. SECTION I. INPUTS FOR LEARNING — SCHOOL LEVEL This section reviews the key school-level inputs necessary for learning in Afghanistan. In order to measure inputs for learning at the school level, we use the following indicators: minimum infrastructure availability, number of classrooms, type of classrooms (permanent, semi-permanent, and temporary), functioning toilets, drinking water availability, boundary wall, security guard at the door, safe shelter, functioning computer, and internet access. Figure 4.1 presents a simple mean comparisons of basic school infrastructure availability by urban/rural school, while Table 4.2 shows the same mean difference comparison by type of school in terms of gender mix. 118 | The Learning Crisis in Afghanistan Figure 4.1: Afghanistan school infrastructure by urban/rural schools Note: The figure shows the percentage of 1) Infrastructure availability, 2) Type of classrooms (permanent, semi-permanent, and temporary) 3) Functioning toilet 4)Drinking water availability 5)Boundary wall 6) Security guard at the door 7) Availability of safe shelter 8) Availability of at least one functioning computer 9)Internet access, and 10) Number of classrooms (permanent, semi-permanent, and temporary),11) Number of students per toilet (defined as the number of students per school by the number of toilets per school) from 170 public schools in Afghanistan. The figures are disaggregated by urban and rural areas. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Despite the surge in reconstruction of schools in Afghanistan through programs such as EQUIP28, only one-third (35%) of public schools have minimum infrastructure availability. We define minimum infrastructure availability as a binary indicator that captures the availability of: (i) functioning toilets and (ii) classroom visibility. Functioning toilets are defined as whether toilets were fit to be used, accessible, clean, and private (i.e. enclosed and with gender separation). To measure classroom visibility, we randomly selected one Grade 4 classroom in which the enumerator placed a printout on the board and checked whether it was possible to read the printout from the back of the classroom. • There is no statistically significant difference in minimum infrastructure availability at the school level among urban and rural schools, or among schools by gender-mix. Schools in Afghanistan are not equipped with sufficient number of classrooms. Less than three- quarters of the schools in the sample (70%) have permanent classrooms (i.e. complete, with floors and 28 Education Quality Improvement Program (EQUIP) is a World Bank project that aims to increase equitable access to quality basic education for students in Afghanistan. EQUIP is implemented by the Afghanistan Ministry of Education and funded by the Afghanistan Trust Fund (ARTF). EQUIP supports building new school infrastructures as well as improving school facilities (e.g. building library, extra classrooms, laboratory, computer purchase, etc.) and teaching materials. Results of the Afghanistan SABER Service Delivery Survey | 119 walls), while 15% have semi-permanent classrooms (i.e. incomplete, plastered without shutters or floor, etc.) and 15% of the schools have temporary classrooms (i.e. classrooms that are made from mud, timber, or held under trees, in a tent, or open-air). Schools in Afghanistan have on average 15 permanent classrooms, 3 semi-permanent classrooms and 4 temporary classrooms, on average. • Urban schools differ significantly from rural schools in terms of the number of permanent classrooms. In particular, urban schools have on average 15 more permanent classrooms for their students as compared to rural schools, while, rural schools have a significantly higher number of temporary classrooms (6 temporary classrooms) compared to urban schools (2 temporary classrooms) (Table 4.1). Table 4.1: Number of classrooms per school by urban/rural schools Overall Urban Schools Rural Schools Mean diff U-R Number of temporary classrooms 4 6 2 4(***) Number of semi-permanent 3 3 3 1 Number of permanent classrooms 15 23 8 15(***) Note: The table shows the number of 1) Permanent, 2) Semi-Permanent, and 3) Temporary classrooms from 170 public schools and total of 3562 classrooms in Afghanistan, disaggregated by urban and rural areas. School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 • Boys schools are more likely to have a higher share of permanent classrooms compared to the rest of the schools. For instance, 85% of boys schools have permanent classrooms, while only 58% of girls schools are equipped with permanent classrooms. Girls schools are more likely to have a higher share of semi-permanent classrooms compared to boys schools and co-ed schools. Girls schools have on average 5 semi-permanent classrooms, while boys schools have only 1 or 2. Lastly, girls schools and school level co-educational schools are more likely to have temporary classroom compared to boys schools and classroom level co-educational schools. Afghanistan schools have severe problems regarding functioning toilets. Overall, almost half (46%) of Afghan schools do not have at least one functioning toilet. Even in schools with such facilities, there are around 169 students per every toilet. Th functioning toilet indicator is defined as a clean, private (enclosed and can be locked), accessible (unlocked, not overflowing) toilet with gender separation as directly observed and verified by the enumerators. • Urban and rural schools are not significantly different in terms of having a functioning toilet in the school. However, there are significantly more students for every toilet in urban schools (229 students per toilet) compared to rural schools (122 students per toilet). • Girls schools are more likely to have functioning toilets compared to boys schools and coed schools at the school level. Nevertheless, the amount of Afghan girls that have to share a toilet is on average 231, which is significantly higher compared to boys schools and coed schools at the classroom level. 120 | The Learning Crisis in Afghanistan Table 4.2: School input availability by gender-mix at the school level (boys/girls/coed schools) Boys Girls COED COED Schools Schools Schools Classroom Mean Total (B) (G) (C1) (C2) Difference Infrastructure 35% 28% 43% 39% 30% None availability Percentage of 15% 10% 17% 20% 14% C1>B* temporary classrooms Percentage of semi- 15% 5% 25% 14% 20% G>B***, G>C1*, permanent classrooms C1>B* Percentage of 70% 85% 58% 66% 67% B>G***, B>C1***, permanent classrooms B>C2* Number of temporary 3.50 1.81 4.24 5.73 1.16 G>B**, G>C2***, classrooms C1>B*,C1>C2* Number of semi- 2.97 1.43 5.58 2.95 2.05 G>B***, G>C1*, permanent G>C2*** Number of permanent 14.97 22.01 14.96 14.88 4.86 B>G*, B>C2***, classrooms G>C2***, C1>C2*** Functioning toilets 46% 40% 61% 44% 38% G>B*, G>C1* Number of Students 169 144 232 201 75 G>B**, G>C2***, per toilet B>C2**, C1>C2*** Drinking water 31% 32% 26% 33% 32% C1>G* Boundary wall 80% 91% 85% 85% 49% B>C2***, G>C2***, C1>C2*** Security guard at the 77% 78% 86% 74% 68% G>C2* door Safe shelter 9% 13% 8% 8% 3% None Functioning computer 41% 48% 55% 43% 12% B>C2***, G>C2***, C1>C2*** Internet Access 5% 0% 13% 2% 0% G>B*, G>C2* Shelter (observed) 4% 10% 4% 1% 0% B>C2* Shelter (not observed) 4% 3% 4% 7% 3% None No shelter 91% 87% 92% 92% 97% None Note: The table shows the percentage of 1) Infrastructure availability, 2) Type of classrooms (permanent, semi-permanent, and temporary), 3) Functioning toilet, 4)Drinking water availability,5)Boundary wall, 6) Security guard at the door, 7) Availability of safe shelter, 8) Availability of at least one functioning computer, 9)Internet access, and 10) Number of classrooms (permanent, semi-permanent, and temporary),11) Number of students per toilet (defined as the number of students per school by the number of toilets per school) from 170 public schools in Afghanistan. The figures are disaggregated by School Type (i.e. Boys only, Girls Only, Coed schools, and Coed classrooms). School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 121 In Afghanistan, 80% of the schools have boundary walls, and 77% have a security guard at the door. • In general, urban schools are better protected than rural schools. There is a statistically significant difference between urban and rural schools in terms of boundary walls and security guard at the school door. For instance, only 12% urban schools do not have any walls surrounding the schools compared to 27% for rural schools. Similarly, 16% of urban schools do not have a security guard at the door, whereas almost one-third of rural school do not have a security guard. • Boys schools, girls schools, and coed schools at the school level do not present any significant difference in availability of security guards or boundary walls. However, coed schools at the classroom level are less likely to have this type of protection. In Afghanistan, 70% of the public schools do not have clean drinking water. A school is classified as having a clean drinking water source if it has a piped water connection, a public tap, a tube-well, a protected well or spring. The main source of drinking water in most Afghan schools is tube well or borehole (40%), followed by protected dug well (15%) and public tap or standpipe (13%) (Figure 4.4). 11% of the schools do not have any source of water available to students (See Figure 4.2) • Rural schools are significantly worse off in terms of their access to drinking water compared to urban schools. Approximately, only 27% of the schools have drinking water available to students in rural areas compared to 36% in urban areas. • Drinking water presents a national challenge regardless of the region and gender make-up of the classroom. Only coed schools at the school level are more likely to have drinking water compared to girls schools. 122 | The Learning Crisis in Afghanistan Figure 4.2: Main sources of drinking water in Afghan schools PIPED WATER into dwelling or household 0.4% 2.4% PIPED WATER to yard/plot or yard connec 1.6% 10.6% 3.2% 9.2% PUBLIC TAP or STANDPIPE 1.1% TUBEWELL or BOREHOLE 1.8% 1.9% PROTECTED DUG WELL 13.3% UNPROTECTED DUG WELL 14.6% PROTECTED SPRING UNPROTECTED SPRING TANKER-TRUCK or CART with small tank/dr 39.7% SURFACE WATER NO WATER SOURCE AVAILABLE PERCENTAGE OF SCHOOLS OTHER Note: The figure breaks down the proportion of schools that have access to drinking water from 170 public schools. A school has a drinking water source if the source is either of the following: (1) piped water into dwelling or household connection; (2) piped water to yard/plot or yard connection; (3) public tap or standpipe; (4) Tube well or borehole; (5) protected dug well; (6) unprotected dug well; (7) protected spring; (8) unprotected spring; (9) tanker-truck; (10) surface water; (11) no water source available. School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 In circumstances that an external attack/hostility on the school or areas near the school occurs, more than 90% on Afghan schools lack available safe shelters to protect students from such incidents. • There is no significant difference in safe shelter availability across rural/urban schools and boys/girls/coed schools. On a national scale, 40% of the schools own at least a computer, but only 5% of these computers are connected to internet. • The rural schools are significantly worse-off compared to urban schools, since 75% of these schools do not have a computer. This share is 40% for urban schools. • Boys schools are better equipped with computers compared to the rest of the schools in the country. Half the schools in Afghanistan are accessible by packed dirt road (unimproved road), followed by 38% of schools that are accessible by tarmac road (See Figure 4.3). Road access is a challenge is many areas in Afghanistan since a significant percentage of students do not live within 15 minutes of distance from their school (See Chapter 5). Improving road conditions can increase students’ enrollment especially in rural communities that are located in remote areas. Results of the Afghanistan SABER Service Delivery Survey | 123 Figure 4.3: Type of roads to access schools in Afghanistan 3.3% TARMAC ROAD 38.4% GRAVEL ROAD 48.9% PACKED DIRT ROAD FOOT PATH ONLY 9.4% PERCENTAGE OF SCHOOLS Note: The figure breaks down the percentage of types of road schools can be accessed from 170 public schools. School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 Box 4.1: Notes from the field on school infrastructure in Afghanistan When the fieldwork was happening, our team was repeatedly approached by teachers, principals, etc. in schools complaining about lack of necessary school inputs such as enough classrooms, proper desks and seats, latrines, clean drinking water, etc. A common problem that almost all principals put forward was lack of latrines and clean drinking water. During one of the notable instances, our team was approach by the principal of a boys school located in Kabul City to discuss lack of functioning toilets. The school had no latrines for 3000 male students. The school principal explained: “We have asked many inspectors who visited our school to address this problem, and I have written to Ministry of Education many times, but we haven’t seen any action taken so far.” While supervising the field work, during a visit in a girls school located in the south of Kabul city, I saw a positive example of community work. I spoke to the school principal, who was a young woman with new perspectives on community participation and girls education. She mentioned the following: “A few years ago parents did not want to send their daughters to school in this neighborhood. There were no girls school that were safe enough for their daughters to attend. After the school building was completed, many parents are now encouraged to send their daughters to school. I even talked to community Shura and we could raise money for a second building in the school site. The community is even providing us with materials and labor.” ____ Contributed by Anahita H Matin 124 | The Learning Crisis in Afghanistan SECTION II. INPUTS FOR LEARNING — CLASSROOM LEVEL Students in general spend a significant amount of their time at school in their classrooms. Classroom environment can have a profound impact on students’ health, engagement, and learning achievements. Classrooms without adequate facilities such as functioning blackboard, lightning, heating, etc. make it difficult to serve students and provide them with a safe, healthy learning space (Jones et al., 2007). In the 2017 Afghanistan SABER SD, a Grade 4 Mathematics or Language classroom was randomly selected and observed/ judged for available classrooms inputs by field enumerators. This section reviews the key inputs necessary for learning at the classroom level in Afghanistan. The following indicators are used to measure inputs for learning at the classroom level: functioning blackboard, classroom visibility, share of students without proper seats and desks, availability of chalk/marker, working electricity, availability of corner library in the classrooms, materials displayed in classroom walls, classroom hygiene level, and share of female/male students wearing uniform. Figure 4.4 and Table 4.3 present simple mean comparisons of classroom infrastructure availability by urban/rural school and by gender-mix categories at the school level, respectively. Classrooms in Afghanistan lack the necessary inputs to provide students with an appropriate learning environment. Overall, 35% of the students do not have proper seats and desks. We define non-proper seats and desks as…. • In terms of seating arrangements, there is no statistical difference between rural (44% without seats/desks) and urban schools (32% without seats/desks). • Girls schools have 40% more students without any proper seats/desks as compared to boys schools. Results of the Afghanistan SABER Service Delivery Survey | 125 Figure 4.4: Classroom and teaching input availability by urban/rural schools Note: The figure shows the percentage of 1) functioning blackboard, 2) classroom visibility, 3) students without proper seats and desks, 4) availability of chalk/marker, 5) working electricity, 6) availability of corner library in the classrooms, 7) whether artworks, charts, etc., displayed in classrooms, 8) classroom hygiene level, 9) share of female vs male students wearing uniform, 10) Lesson plan observed and available, 11) Share of students with textbooks, collected from a randomly selected grade 4 classroom in 170 public schools in Afghanistan, disaggregated by urban and rural areas. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Working electricity in the classroom is another input that most schools lack. Specifically, less than one-third of public schools (29%) do not have electricity available in the classroom. • Urban schools are 18% more likely to have working electricity in the classroom compared to rural schools. In particular, 33% of urban schools have working electricity compared to only 15% for rural schools. 126 | The Learning Crisis in Afghanistan • In general, girls schools are better equipped with electricity compared to coed schools, however they are not significantly different from boys schools. On a positive note, around 89% of the classrooms are equipped with a functioning blackboard. A blackboard is considered functional if the text written on the board could be read from both the front and back of the classroom, and if there is chalk available to write on the blackboard. In particular, 99% of the schools have chalk and/or markers available in the classrooms. Table 4.3: Classroom and teaching input availability by school type (boys/girls/coed schools) Total Boys COED Schools Girls Schools COED (B) Schools (G) (C1) Classroom (C2) Mean Difference Functioning blackboards 89% 92% 76% 95% 97% C2>G* Classroom visibility 79% 74% 73% 86% 79% None No proper seats and 35% 19% 57% 30% 51% G>B***, C2>B**, desks G>C1* Chalk/marker for 99% 98% 100% 100% 97% None blackboard Working electricity 29% 35% 50% 13% 9% G>C1***, G>C2*** Corner library 1% 0% 0% 2% 0% None Materials displayed on 34% 22% 50% 36% 1% G>C2***, C1>C2** walls Hygiene: Extremely clean G>B*, C2>B**, 8% 1% 16% 5% 26% and well maintained C2>C1* Hygiene: Reasonably 79% 78% 79% 84% 56% C1>C2** clean and maintained Hygiene: Not very clean 13% 20% 5% 11% 18% None or maintained G>B***, G>C1***, Share of boys in G>C2, 64% 20% 100% 79% 29% uniforms C1>B***, C1>C2** B>G***, B>C1***, Share of girls in uniforms 50% 100% 23% 32% 43% B>C2*** Lesson plan (observable) 71% 56% 84% 77% 55% G>B*, G>C2** Students with textbooks 86% 85% 87% 86% 75% None Note: The table shows the percentage of 1) functioning blackboard, 2) classroom visibility, 3) students without proper seats and desks, 4) availability of chalk/marker, 5) working electricity, 6) availability of corner library in the classrooms, 7) whether artworks, charts, etc., displayed in classrooms, 8) classroom hygiene level, 9) share of female vs male students wearing uniform, 10) Lesson plan observed and available, 11) Share of students with textbooks, collected from a randomly selected grade 4 classroom in 170 public schools in Afghanistan, disaggregated by School Type (i.e. Boys only, Girls Only, Coed schools, and Coed classrooms). School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 127 Around 80% of classrooms in Afghanistan are sufficiently visible to the students. As mentioned earlier, to measure classroom visibility, we randomly selected one Grade 4 classroom in which the enumerator placed a printout on the board and checked whether it was possible to read the printout from the back of the classroom. • Urban schools do not differ much in classroom visibility from rural schools. • Public schools do not differ in classroom visibility in terms of gender-mix at the school level. Corner libraries in the classrooms is practically non-existent in Afghanistan, with 1% of urban classrooms equipped with such facility. On average, one-third of Afghan classrooms displayed some kind of educational material on the walls, such as artwork, charts, maps, etc. • Rural schools have a significantly lower share of educational material displayed on the wall (14%) as compared to urban schools (40%). • Girls schools and co-ed schools at the school level are more likely to have educational material displayed on the walls compared to co-ed schools in the classroom, which are generally located in rural areas. In terms of classroom hygiene, 8% of the classrooms were extremely clean and well maintained, 80% were reasonably clean, and 13% were not very clean or well maintained. • Urban schools are not very different from rural schools in terms of classroom hygiene. • Girls only schools have significantly higher share of classrooms that are extremely well maintained (16%) vs boys schools where only 1% of classrooms are extremely well maintained. Coed schools in the classroom are also better maintained compared to boys schools and coed schools at the school level. Around half of Grade 4 Afghan students wear uniforms to go to schools. The share of boys in uniform (64%) is slightly higher than the share of girls in uniform (50%), but this difference is not statistically significant. The share of girls and boys who are wearing uniforms is also not significantly different from one another in rural/urban schools. Inputs for Teaching (Teaching Equipment) Textbooks are the most basic and necessary learning aid used in the classrooms. In an environment like Afghanistan which suffers from extremely limited access to computer and internet technology available for students, simple use and exposure to textbooks can significantly improve the quality of learning in classrooms. Evidence from countries such as India, Brazil, etc. also suggests that teachers who are prepared in the classroom and plan structured lessons can significantly impact student performance (WDR, 2018). 128 | The Learning Crisis in Afghanistan In order to measure basic inputs necessary for teaching in the classrooms in Afghanistan, two indicators are used: (i) the share of students with textbooks and (ii) availability of a lesson plan. Overall, approximately 86% of Grade 4 Afghan students have textbooks available. This indicator measures the number of language and mathematic textbooks depending on the subject that the students use in the classroom divided by the number of students present in the classroom. • There is no statistical difference in textbook availability between urban and rural schools. • Textbook availability does not differ across gender-mixes at the school level. Almost three-quarters (71%) of teachers had their lesson plan ready and available. • Rural schools are not very different from urban schools with respect to the readiness of the lesson plan from the teachers. 74% of urban schools had their lesson plan ready compared to 63% in rural schools. • Teachers in girls schools were more likely to have the lesson plan ready and available (84%) compared to boys schools (56%) and coed schools in the classroom (55%). Girls schools are not very different than coed schools at the school level. SECTION III. SCHOOL INPUTS — INTERNATIONAL COMPARISON In terms of inputs at the school level, Afghanistan is comparable to the average SSA countries. Table 4.4 shows how Afghanistan compares to other SDI countries in terms of school and classroom inputs availability. The minimum infrastructure availability (i.e. functioning toilets and classroom visibility) in Afghanistan (35%) is comparable to the average SSA countries at 36%. This puts Afghanistan above countries like Togo (14%), Nigeria (13%) and Mozambique (29%) but below Uganda (57%), Kenya (60%), and Tanzania (36%). The proportion of Afghan schools with functioning toilets and drinking water is significantly lower compared to Mozambique, Kenya and Laos. Afghan schools have a significantly higher share of students with textbooks compared to African countries, but they are comparable to Laos. For instance, 37% of SSA countries have textbooks available for their students, whereas 86% of the students in Afghanistan have textbooks. Classrooms in Afghanistan seem to be slightly better equipped than Mozambique and Kenya in terms of chalk and marker availability in the classroom, but they are very similar to Laos classrooms in this respect. Also, an equal proportion of Afghan and Laos teachers (71%) seem to have their lesson plan ready and available. The share of classrooms with a functioning blackboard in Afghanistan is lower compared to SSA countries. In SSA countries, practically all classrooms have a functioning blackboard, while in Afghanistan around 10% of classrooms still do not have a functioning blackboard. Results of the Afghanistan SABER Service Delivery Survey | 129 Table 4.4: School and classroom inputs availability — international comparison SDI Laos Afghanistan Average Togo Nigeria Uganda Tanzania Mozambique Kenya PDR 2017 (SSA) 2013 2013 2013 2014 2014 2012 2017 School Inputs Minimum 35 36 14 13 57 36 29 60 58 Infrastructure availability Functioning 46 - - - - - 65 74 98 toilets Drinking water 31 - - - - - - - 72 source Working 29 - - - - - - - 29 electricity Classroom 79 - - - - - 92 86 84 visibility Classroom inputs Functioning 89.2 99.2 - - - - 99 99 100 blackboards Chalk/marker to 99.2 - - - - - 96 93 99.7 write on the blackboard Students with 86 37 76 34 6 26 68 45 84 textbooks Lesson Plan 71 - - - - - - - 72 (available and observable) Note: The table shows the percentage of 1) Infrastructure availability, 2) Functioning toilets, 3) Drinking water availability, 4) Functioning blackboards, 5) Availability of chalk/marker to write on the blackboard, 6) Classroom visibility, 7) Working electricity, 8) Share of students with textbooks, 9) Lesson Plan Available and observable in Afghanistan compared to countries in Sub-Saharan Africa. All individual country statistics are calculated using country-specific sampling weights. The Average for SSA countries can be seen in SDI Average (Sub-Saharan Africa) column. SDI Average (Sub-Saharan Africa) is taken by averaging over the SSA country averages, including Mozambique, Kenya, Nigeria, Senegal, Tanzania 2014 and 2011, Togo, and Uganda. The figures from SDI Average (Sub-Saharan Africa) and Mozambique are taken from the Mozambique 2014 SDI Educational Technical Report. The figures from Kenya are taken from the Kenya 2012 SDI Educational Technical Report. The figures from Loa PDR are taken from the Loa PDR SABER Service Delivery 2018 Technical Report. 130 | The Learning Crisis in Afghanistan Chapter 5 : Student Support Supportive home and school environments can be instrumental in allowing students to focus on their responsibilities at school and be more dedicated towards learning (WDR 2018). A stable, enabling environment at home, educated surroundings, teachers who motivate their students, availability of basic equipment like pencils and exercise books, all ensure that students are able to focus more on learning and less on the obstacles in the way of learning. The main finding of this chapter is that Afghan students do not have the necessary support from home and school to allow them to be fully prepared to learn in school. Despite the severe lack of an enabling environment, these students continue to show resilience and are enthusiastic about going to school, as observed by their attendance and engagement levels in school. Do Afghan students receive the necessary support from school? A supportive environment at school offers resources that facilitate learning. Students cannot be expected to learn if they do not have the necessary equipment with which to study. In Afghanistan, only about one-third (36%) of all schools have the minimum equipment required in classroom to allow students to study, which includes having a functional blackboard, an exercise book and pens or pencils. This is considerably low even when we compare it to Sub-Saharan African (SSA) countries, where the figure is closer to 60%. Surprisingly, rural schools are much better equipped with minimum equipment availability compared to urban schools. There is no statistically significant difference in minimum equipment availability between schools in terms of gender mix. Schools with special classes for students with special needs are substantially scarce in Afghanistan. A third of the Afghan schools (31%) report having special needs students; however, only 2% of the schools offer special classes for these Results of the Afghanistan SABER Service Delivery Survey | 131 students. Moreover, only 18% of Afghan schools report offering services for students to cope with Post-Traumatic Stress Disorder (PTSD). Urban schools and same-sex schools are much better equipped with special needs facilities and PTSD support than rural schools and co-educational schools. Lastly, only half of the teachers provide a supportive environment to students in the classroom in terms of inviting the student to the blackboard, checking his performance individually, or calling the students by their names. Do Afghan students receive the necessary support from home? A supportive environment at home necessitates families narrow down the responsibilities of students to learning only. Students in Afghanistan do not have a strong support system provided by their home and families. Less than half (40%) of Grade 4 students in Afghanistan have home support available. While almost all students had breakfast during the morning of the survey and had missed less than 5 days of school that month, only 50% of students said that they have someone to help them with homework. This is reflected in the fact that only a small percentage of students have parents that have some level of education. Half of all students reported that their father is illiterate, and around 75% reported that their mother is illiterate. Family support in Afghanistan does not vary significantly by urban or rural areas or by type of school in terms of gender-mix. Are Afghan students engaged with learning in the classroom? Being present at school and engaged in the classroom paint a picture of the students’ dedication to learning. Around a quarter of the students were observed to be absent on average. Once in the classroom, Afghan student do not tend to misbehave, and are engaged most of the time. This chapter presents the different types of support available to students. Section I describes student support provided in school, Section II delves into the student support available in their homes, and Section III explores the student dedication to learning. For the case of this chapter, it is important to point out that the Afghanistan SABER SD could not cover the cost of a full survey of student household conditions, therefore the data for this section is sparser and the discussion more tentative. SECTION I. STUDENT SUPPORT PROVIDED IN SCHOOL Students in Afghanistan do not have the necessary support from their schools. In order to measure student support at the school level, three types of indicators are used: (i) minimum school equipment availability, (ii) physical and/or socio-emotional support, and (iii) supportive environment in the classroom. Minimum School Equipment Availability Only one-third (36%) of the classrooms in Afghanistan have minimum school equipment availability, despite the fact that 90% of the schools contain a functioning blackboard. Minimum school equipment availability is a binary indicator that captures the availability of: (i) functioning blackboard and chalk and (ii) pens, pencils and exercise books in Grade 4 classrooms. A blackboard is considered functional if the text written on the board could be read from both the front and back of the classroom, and also if there is chalk available to write on the blackboard. We considered that the classroom met 132 | The Learning Crisis in Afghanistan the minimum requirement of pens, pencils and exercise books if both the share of students with pen or pencils and the share of students with exercise books were above 90%. Figure 5.1 shows the breakdown of the equipment used in class by urban and rural schools. Rural schools are much better equipped with minimum equipment availability compared to urban schools. Around 50% of rural school have at least a functioning blackboard, chalk, pens, pencils, and exercise books for the students. Only one-third of urban schools have minimum school equipment availability (Figure 5.1). Figure 5.1: Minimum school equipment availability by urban/rural school (%) Note: This table shows the percentage of students with 1) a pen or pencil, 2) an exercise book, and 3) Minimum Equipment Availability in a randomly selected grade 4 classroom from 170 schools in Afghanistan. The figures are disaggregated by urban and rural areas. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Almost all Afghan students (90%) have a pen or pencil to work with, and more than half (59%) are equipped with an exercise book. However, less than half of the students have both an exercise book and a pen/pencil. A student needs materials and resources from the school to be an attentive and productive learner. This includes having functional blackboards with chalk available to observe what the teacher is demonstrating in class, exercise books in the classroom to take down notes or solve problems, and pens or pencils to work with (Figure 5.1).Students in urban schools do not differ from those in rural schools in terms of pens/pencils availability. However, students in rural schools are significantly more likely to have exercise books compared to those in urban schools. There is no statistically significant difference in minimum equipment availability between schools in terms of gender mix. Results of the Afghanistan SABER Service Delivery Survey | 133 Figure 5.2: Minimum school equipment availability by type of school (%) Note: This table shows the percentage of students with 1) a pen or pencil, 2) an exercise book, and 3) Minimum Equipment Availability in a randomly selected grade 4 classroom from 170 schools in Afghanistan. The figures are disaggregated by urban and rural areas. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Box 5.1: Comparing Availability of Learning Equipment Internationally Afghanistan students had a higher percentage of pen/pencil availability compared to the Sub-Saharan Africa average. However, in terms of the minimum equipment available, Afghanistan classrooms showed below average statistics. Only 35% of the classrooms in Afghanistan had the minimum equipment available, i.e. (1) at least 90% of students with a pen/pencil and an exercise book, (2) a functioning blackboard and chalk. Table 5.1: Comparing availability of learning equipment internationally SSA Afghanistan (Average) Mozambique Kenya Tanzania Togo Pen/pencil availability (% students) 92 82 97 98 95 19 Minimum equipment availability (% 36 58 77 74 62 24 classrooms) Note: The table presents the Pen/pencil availability and Minimum Equipment Availability in Afghanistan compared to other countries in Sub-Saharan Africa. All individual country statistics are calculated using country-specific sampling weights. The average for all SSA countries, reported under the heading “Sub-Saharan Africa (Average),” is taken by averaging over the SSA country averages, including Mozambique, Kenya, Nigeria, Senegal, Tanzania 2014 and 2011, Togo, and Uganda. The figures from Sub Sahara Africa are taken from the Mozambique 2014 SDI Educational Technical Report 134 | The Learning Crisis in Afghanistan Physical and Socio-Emotional Support Facilities provided by the school that offer physical, psychological, or emotional support can narrow the social gaps by including disadvantaged members of the community. Children in conflict zones are more likely not to be in school and to drop out of school than those living in non-conflict zones (WDR 2018). In a war-torn country like Afghanistan, offering support to students suffering from Post- Traumatic Stress Disorder (PTSD) provides them with a means of coping with stress and trauma which would not only improve learning, but also overall life outcomes. Sustained stress or trauma can result in lowered ability to learn, retain, and process information. In order to measure whether Afghan students have physical or socio-emotional support, we analyze whether schools have students with special needs and whether schools offer special classes for these students, as well as PTSD support. Schools with special classes for students with special needs are substantially scarce in Afghanistan. A third of Afghan schools (31%) report having special needs students; however, only 2% of the schools offer special classes for these students. Almost all schools with special needs students (96%) do not have trained teachers to handle classes with special needs student. Urban schools are better prepared than rural schools for coping with students with special needs. Urban schools do not differ from rural schools in terms of proportion of students with special needs. However, the proportion of classrooms/teachers for special needs students is significantly higher in urban schools (4%) compared to rural schools where it is nonexistent (Figure 5.3). Figure 5.3: Students/Classrooms with special needs and PTSD support by urban/rural school Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. About 18% of Afghan schools report offering services for students to cope with Post-Traumatic Stress Disorder (PTSD). Only 20% of these schools referred the students to a mental health professional. The majority of the cases consist of access to an untrained professional, such as one-on-one counseling with a teacher (47%), or group therapy with a teacher (Figure 5.3). Results of the Afghanistan SABER Service Delivery Survey | 135 Generally, urban schools are much better equipped with special needs facilities and PTSD support than rural schools. Figure 5.3 shows that one-third (33%) of Afghan schools are prepared to offer PTSD support to students compared to only 6% of rural schools. Co-educational schools have significantly less PTSD support for students compared to same-sex schools (Figure 5.4). Figure 5.4: Students/Classrooms with special needs and PTSD support by type of school Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Supportive Environment in the Classroom Afghanistan’s classrooms show moderate support provided by teachers to students in the classrooms. In order to measure whether Afghan students have supportive environment in the classroom we use three different measures: (i) whether the student writes on the blackboard (ii) whether the teacher visits the students individually and (iii) whether the teacher calls students by their name. We also analyze whether the supportive environment in the classroom varies by teacher’s gender and student’s gender. 136 | The Learning Crisis in Afghanistan Figure 5.5: Supportive behavior in the classroom by urban/rural schools Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Around half of the teachers provide a supportive environment to students in the classroom. In almost 55% of the classrooms observed, students go and write on the blackboard. Teachers visit students individually during class in about 66% of the classrooms. In 46% of the classrooms observed, teachers called students by their name (Figure 5.5). Teachers in urban schools are more likely to be more supportive with their students compared to those in rural schools, especially in terms of visiting the students individually (Figure 5.5). Generally, teachers support behavior is similar across types of schools in terms of gender mix (Figure 5.6). Nevertheless, teachers in co-educational schools in the classroom are more likely to call the students by their names probably due to the small size of the classrooms in comparison to the other types of schools. Results of the Afghanistan SABER Service Delivery Survey | 137 Figure 5.6: Supportive behavior in the classroom by school type in terms of gender mix Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Female teachers are more likely than their male counterparts to create a supportive environment to their students, especially by visiting students individually. For instance, 75% of the female teachers visit their students individually compared to 61% for male teachers. However, female teachers do not differ much from male teachers in terms of calling their students by their name or inviting students to write on the blackboard (Figure 5.7). Figure 5.7: Supportive behavior in the classroom by teacher gender Source: SABER Service Delivery - Afghanistan 2017 In co-educational schools in the classroom, boys are significantly more likely to be called to the blackboard, or visited individually by teacher, while girls are more likely to be called by their names. This could reflect cultural norms of not asking girls to actively participate in the classroom (Figure 5.8). 138 | The Learning Crisis in Afghanistan Figure 5.8: Supportive behavior in co-education classrooms by student gender Note: These figures show the percentage of schools where the 1) Students wrote on the blackboard, 2) Teachers visited students individually, 3) Teacher called students by their names. Observations are made from a randomly selected grade 4 classroom in 170 public schools in Afghanistan. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Results of the Afghanistan SABER Service Delivery Survey | 139 Box 5.2: What does a typical student in Afghanistan look like? An average a Grade 4 student in Afghanistan is between 10 and 11 years old and has 6 siblings, on average. Students in rural areas are slightly older than in urban areas. They are likely to have an illiterate mother and are as likely to have an illiterate father as a literate one. Specifically, 53% of the students have an illiterate father, while 73% of the students have an illiterate mother. Parents in rural areas are more likely to be illiterate compared to those in urban areas. The average student is more likely to speak Dari at home, but in some areas, they also speak Pashtu. The majority of the students are likely to have electricity and running water at home, however this is significantly higher for students in urban areas. Only half of the students have a bed at home. The average student has a mobile phone at home, but not a computer. Both assets are more frequently available to students living in urban areas. Table 5.2: Background characteristics of Afghan students Urban-Rural Overall Urban Rural mean diff Age 10y 9m 10y 6m 10y 11m -5m*** Percentage of boys 50% 46% 54% -8% Number of siblings 6 6 6 0 Percentage with illiterate father 53% 46% 59% -13%*** Percentage with illiterate mother 73% 68% 78% -10%*** Language Dari 59% 62% 57% 4% Pashtu 26% 27% 25% 2% Other 14% 11% 17% -6% Dwelling characteristics Electricity 82% 88% 76% 13%*** Running water 75% 82% 69% 13%*** Assets Student has a bed 51% 55% 48% 7% Student has a mobile phone 78% 84% 74% 10%** Student has a computer 31% 41% 22% 19%*** Note: This table shows the summary statistics of the average grade four student in Afghanistan, disaggregated by urban and rural, and shows any significant differences between the two. Variables include age, gender, number of siblings, parents’ education, language spoken at home, dwelling characteristics, and household assets. Collected from a randomly selected grade 4 classroom of students from 170 public schools in Afghanistan. School level weights are used to compute these numbers. Significant differences are stated in bold. Levels of significance are depicted by asterisks. (1%=***, 5%=**, 10%=*). Source: SABER Service Delivery - Afghanistan 2017 140 | The Learning Crisis in Afghanistan SECTION II. STUDENT SUPPORT AVAILABLE FROM HOME In order to measure students’ support available from the family background, we use two types of measures: (i) socioeconomic background, measured in terms of family structure, parental education, household assets, and distance to school, and (ii) home support, measured in terms of whether the students have had breakfast, have someone at home to help with homework, and whether they have had to take days off from school. Most Afghan students (75%) in Grade 4 live with both their parents. Very few students reported living with someone other than both their parents. Family background, such as parents’ education level, socioeconomic status, and household assets is considered to be one of the main predictors of learning outcomes (WDR, 2018). Figure 5.9: Who Do the students live with? — Afghanistan Note: This figure breaks down the family structure of the students from 170 public schools in Afghanistan, disaggregated by gender. School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 In general, Afghan students have illiterate parents with mothers being significantly more illiterate than fathers. In particular, half of the students (53%) reported having an illiterate father, and around 73% reported having an illiterate mother. Both parents in rural areas are significantly more likely of being illiterate compared to those in urban areas (Figure 5.10). Results of the Afghanistan SABER Service Delivery Survey | 141 Figure 5.10: Parental education in Afghanistan Note: This figure shows the parents’ education levels for students observed in a randomly selected classroom from the 170 public schools in Afghanistan. School level weights are used to compute these numbers. Significant differences are stated in bold. Levels of significance are depicted by asterisks. (1%=***, 5%=**, 10%=*) Source: SABER Service Delivery - Afghanistan 2017 Afghan students living in urban areas are more likely to live in wealthier families compared to those in rural areas. In particular, they are more likely to have mobile phone, books, refrigerator, television, toilet, a metal roof, and walls in their dwelling. Students in rural areas are more likely to have animals such as a goat, sheep, etc. (Figure 5.11). 142 | The Learning Crisis in Afghanistan Figure 5.11: Household assets by urban/rural schools Note: This figure breaks down the students’ household assets by urban/rural schools. Collected from a randomly selected grade 4 classroom of students from 170 public schools in Afghanistan. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Long commutes can prevent students from making the trip to school, especially in the case of girls whose families may fear harm to their safety or chastity (Lloyd et al. 2005; Sutton 1998). Afghan students reside around 15 minutes away from their school, which can be even less in urban areas and at most 30 minutes in rural areas. Walking is the most commonly used method of commuting to school with 80% of students ascribing to it (Figure 5.12). Results of the Afghanistan SABER Service Delivery Survey | 143 Figure 5.12: Distance from school Note: This figure shows the average time in minutes that it takes Grade 4 students to get to school by urban/rural school. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. Less than half (40%) of Grade 4 students in Afghanistan have home support available. We measure home support as the percentage of Grade 4 students that (i) have breakfast on the day of the survey, (ii) receive support from their parents or siblings to complete their homework and (iii) have missed less than 5 school days last month (Figure 5.13). Figure 5.13: Home support by urban/rural schools Note: This figure breaks down the Home Support indicator, by showing what percentage of students 1) Had breakfast that morning, 2) Have a parent/sibling to help with homework, 3) Missed less than 5 days of school in the last month, and finally 4) have Home Support (i.e. meet all 3 conditions). Collected from a randomly selected grade 4 classroom of students from 170 public schools in Afghanistan. School level weights are used to compute these numbers. The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017. The data in Figure 5.13 indicate that while almost all students had breakfast during the morning of the survey and had missed less than 5 days of school that month, only 50% of students said that they have someone to help them out with homework. Family support in Afghanistan does not vary significantly by urban or rural areas, or by type of school in terms of gender-mix. 144 | The Learning Crisis in Afghanistan SECTION III. STUDENT DEDICATION TO LEARNING In order to measure student dedication to learning we use the following indicators: (i) student absence rate, (ii) student engagement in class, (iii) time spent learning by an average student per day (in hours). Table 5.3 shows student absence rate and student dedication to learning for using data from the Afghanistan SABER SD Survey. Student absence rate Around a quarter (23%) of Afghan students were observed to be absent, on average, from a randomly selected Grade 4 classroom (Table 5.3). This number seems quite high compared to the figures from PISA 2012 where 15-year old students were asked about their attendance in the last two weeks. 18% of students reported skipping classes at least once, while 15% said they had skipped a day of school in the given period (OECD, 2014). This would be equivalent to 1.5% absence rate. 29 Box 5.3: Notes from the Field High Absence rates While supervising the field work in Kabul city, our team had the opportunity to meet and talk with many school teachers. Student absence was an issue addressed by both teachers and principals. During a boys school visit in Kabul city, we noticed half-empty classrooms. When we inquired, teachers and principals told us: “Some of these boys have lost their fathers or elder brothers due to attacks or other reasons, and they are the sole breadwinner for their families. They have to work half a day and come to school the next half. Many of them are too tired to listen in the class. We still ask their families to send them to school. We want them to learn and become valuable citizens.” Our field team in Sari Pul province shared a similar story with us from the principal of a school located in a rural district: “As you see people in this village are poor. Most parents are illiterate. Some cannot afford pencils and notebooks for their children. Most of the people are dependent upon their children’s labor to work on their farms or herd their livestock. It is the harvest season here and we do not see many students show up at school. They are helping their families.” ____ Contributed by Field Team Student Engagement in Class We measure 3 student engagement indicators with classroom observation instruments: (i) Student Off-task rate using the Stallings instrument, (ii) Student Engagement using the CLASS instrument, (iii) Behavior Management using the CLASS instrument. The overall number of Afghan students who were found to not be paying attention, therefore considered “off-task” was only about 3% (Stallings Method). The Stallings method involves measuring classroom activity 10 times for the duration of a lesson in a randomly selected Grade 4 lecture. 29 These results are self-reported and actual absence rates may be different if use the same methodology we use in this survey. Results of the Afghanistan SABER Service Delivery Survey | 145 Observers mark the general activity of both the teacher and the student in the snapshots, and the size of the group of students engaged in that activity. We use this instrument to distinguish between on- task and off-task activities for students. The percentage of time students were engaged in off-task activities out of the total allotted time of the class is used to generate the Student off-task rate. We use this to arrive at an estimate of students’ attentiveness in class. Student Engagement in Afghanistan is scored on the high end of the scale, meaning that students were actively engaged in the classrooms (CLASS method). The CLASS student engagement construct measures the extent to which students are participating in the learning exercises taking place in the class. It distinguishes between disengagement, passive engagement, and active engagement. At the high end of the spectrum, students are actively engaged in the class (Figure 5.14). Afghan students scored on the lower end of the behavioral engagement scale, meaning that no misbehavior was observed in the classroom (CLASS method). The Behavior Management scale measures any student misbehavior in the class. This again points to little time being lost in activities that are not related to learning. In summary, students behave well and were actively engaged. Note that this may not be reflective of teacher skills but rather cultural norms of students, as discussed in chapter 2. Figure 5.14: Student engagement using CLASS Note: This figure displays the average score of the CLASS Student Engagement domain, on a scale of 1 to 7 with 7 being the highest. 88 randomly selected Grade 4 classrooms out of 170 were observed using the CLASS instrument. Source: SABER Service Delivery - Afghanistan 2017 Time spent learning by an average student per day Students spend about 1 hour 43 minutes of their time in school engaged in learning activities each day. The average scheduled teaching day, excluding break times, averaged 3.5 hours in Afghanistan. Time spent learning per day is measured by the time spent by an average student receiving instruction from a teacher. The teaching time is adjusted for the time teachers are absent from the classroom and students are absent from school, on average, and for the time that students are off-task while the teacher teaches. The following formula is used to calculate this indicator: Time spent learning = {[1- Class absence]*[Time on task * (1-Student absence) * (1-Student off task)]} * [Scheduled school duration] Therefore, even in the case of a low time off-task and high student engagement, students may not learn much simply because they are spending too few hours at school. 146 | The Learning Crisis in Afghanistan Table 5.3: Absence rate and student dedication to learning Overall Urban Rural Diff Student absence rate 23% 22% 28% -6% Percentage of students out of total who are off-task 3% 2% 4% -2% Time spent learning by an average student per day (in hours) 1h 43m 1h 43m 1h 45m -2m Note: This table shows down the Student Effort and Dedication indicator: 1) Student Absence Rate which is measured using the classroom observation data. It is measured as the percentage of the number of students absent from the class out of all the students registered in the class, 2) Students Off-Task which is an indicator that measures the percentage of time students do not pay attention. It uses the classroom observation in one randomly selected Grade 4 classroom. It is computed as the average percentage of students who are off-task which is measured 10 times in the Stallings tool through the entire duration of the lesson, 3) Time Spent Learning per day which adjusts the length of the school day by the share of teachers that are present in the classroom, on average, and students present in school, on average, and for the time that students are on-task while the teacher teaches using the classrooms observations data. Collected from a randomly selected grade 4 classroom of students from 170 public schools in Afghanistan. School level weights are used to compute these numbers. Significant differences are stated in bold. Levels of significance are depicted by asterisks. (1%=***, 5%=**, 10%=*). Source: SABER Service Delivery - Afghanistan 2017 Box 5.4: Comparing Student Effort Indicators Internationally Afghanistan students had a higher student absence rate compared to Laos, but lower than that in Mozambique. This student absence rate is measured by observing the number of students absent from a Grade 4 classroom, out of the total number of students registered in the class. Afghani students outperformed Mozambique students by being off-task only about 2% of the time compared to 10% in Mozambique. However, it should be noted that the off-task rate is calculated using the Stallings method in Afghanistan, whereas in Mozambique it is calculated using the SDI CLASS instrument. Both instruments require observers to mark students as off-task or uninvolved, but while Stallings measures the classroom activity 10 times during the entirety of the lesson (every 3-4 minutes in Afghanistan), the SDI tools require observers to mark activity every 5 minutes during the lesson (which lasts around 45 minutes on average in Mozambique). Table 5.4: Student effort indicators internationally Availability of Inputs (%) Afghanistan Mozambique Laos Student Absence Rate (%) 23 56 9^ Students off-task (%) 2 10 N/A Note: The table presents the student absence rate and students off-task rate compared to Mozambique and Laos. All individual country statistics are calculated using country-specific sampling weights. The figures for Mozambique are taken from the Mozambique 2014 SDI Educational Technical Report. ^Figures for Laos and taken from the Delivery of Education Services in Lao PDR report Results of the Afghanistan SABER Service Delivery Survey | 147 Chapter 6 : Community-Based Education in Afghanistan Community-Based Education (CBE) schools were developed during the 1990s when the public-school system had collapsed and home-based education was the only option for girls schooling under the Taliban. The Ministry of Education (MoE) recognizes the community-based approach to education as an alternate pathway for improving access to education. By definition, a CBE school is a MoE outreach school/class, and is jointly established by the MoE, communities, and facilitating partners (NGOs). They can also be jointly established by the MoE and the community in remote, rural, and sparsely populated areas (villages) where: (i) a gender appropriate public school for children does not exist, (ii) children live at a walking distance of more than three kilometers from a public school, and (iii) a significant number of children have missed the opportunity of formal education due to insecurity, distance, lack of female teachers, lack of learning materials and supplies, have crossed the school age, or require accelerated learning opportunities. Because there are more public schools for boys than for girls in rural areas, CBE schools are usually created for girls. Although policy requires NGOs to establish CBE schools at least three kilometers away from public schools, many NGOs use the fact that there are no girls schools as a rationale for setting up CBE classes for girls right next to public schools for boys. A CBE school provides education for children from grades 1 to 9 and is an integral part of general education in the country. This chapter describes the CBE schools in Afghanistan and compares their performance to the public schools in terms of learning outcomes and the four key school level ingredients for learning, according to the WDR (2018): effective teaching, learning focused school inputs, prepared learners, and skill management and governance that pulls them all together. The 2017 Afghanistan SABER SD surveyed 30 CBE schools located in the provinces of Ghor, Paktia, and Khost, where a total of 672 Grade 4 students were assessed. Similar to the public school analysis, the structure of the student assessment was designed not only to test Grade 4 students’ overall skills in Language and Mathematics, but also to assess whether their performance was aligned with the learning structure and curriculum in Afghanistan. Results of the Afghanistan SABER Service Delivery Survey | 149 How do CBE school students differ from public school students in learning outcomes? The first main finding of this chapter is that students in CBE schools tend to significantly outperform those in public schools, both in Language and Mathematics. Despite this good news, the overall learning performance for CBE schools is also worrisome as more than half of Grade 4 CBE students, in both Language and Mathematics, are still performing significantly below their current grade level. In particular, students in CBE schools do not only master better the Grade 1 curriculum, they are especially better performing in Grade 4 tasks with a higher level of difficulty. In Language, the learning gap is found in harder Grade 4 level questions involving the correct use of grammar, tenses, and reading comprehension. Similarly, in Mathematics, the biggest difference is attributed to harder questions such as word problems, identifying shapes, understanding fractions and division, and double digit subtraction and multiplication. The public-CBE learning gap in Mathematics is larger than the one in Language and can range between 15 to 20 percentage points. In line with these results, we also find that CBE Afghan students are, on average, half a curriculum-adjusted year of schooling (0.5 years) ahead of public school students. A significantly higher proportion of CBE students are able to perform questions pertaining to Grade 4. What explains these results? While we do not have data to answer this question rigorously, we apply the WDR 2018 framework to understand the degree to which there are differences in the quality of the service delivery for CBE versus public schools. We find that CBE schools have worse infrastructure and equipment with only 13% of the schools having functioning toilets (versus 46%), 38% having electricity (versus 80%) and only 14% of the students having textbooks (versus 79%). Furthermore, CBE schools have younger teachers (28 vs 36 years old) with less experience (3 vs 12 years), and less formal training. While CBE teachers have on average higher salaries (140 vs 100 USD), they are more prone to experience salary delays (almost 40 percentage points higher). On the positive side, CBE schools have smaller class sizes than public schools (27 vs 44 students) and lower student absence rate (13% vs 23%). Teacher subject and pedagogic content knowledge cannot explain the difference in CBE students outperforming public school students as public school teachers outperformed CBE School teachers in Mathematics (3.1 curriculum adjusted years of schooling vs 2.4) and have a similar performance in language with CBE teachers outperforming public school teachers in the reading comprehension task (72 vs 44), underperforming in the Cloze Task (44% vs 32%). Similar results are also observed in pedagogy knowledge. However, CBE teachers did display significantly higher skills in the classrooms. For example, 90% of CBE teachers explained the topic of the lesson at the start and summarized what was learned at the end (versus 54%), and only around 5% of the lesson seemed unplanned to the observers (versus 15%). During their lessons, many more CBE teachers asked questions that required students to recall information or to practice what was learned as well as other questions that required higher order skills and encouraged students to apply what was learned to different contexts and be creative. Overall, 67% of CBE teachers mixed lower and higher order questions in their class (versus 33% of public school teachers). In response to students’ answers, around 40% of CBE teachers consistently gave positive feedback and corrected mistakes without scolding students (versus 24% of public school teachers). Overall, 3 in 10 CBE teachers (versus less than 2 in 10 public school teachers) 150 | The Learning Crisis in Afghanistan apply the full set of beneficial skills to promote learning—structuring, planning, asking questions, creating a positive environment, and providing constructive feedback—in their lessons. CBE teachers’ support system replicate the findings on inputs discussed above; teacher material was not available for CBE schools. Professional development focuses more on content knowledge in CBE schools, and teachers have less autonomy to choose which teaching methods to use which helps to explain why they display better skills in the classroom. In summary, while we cannot know for certain whether the better skills teachers display in the classroom are behind the relative better performance of CBE students, these findings provide the rationale for more research on this topic. In the sections that follow, we describe in more details the answers to the questions raised above: How do CBE school students differ from public school students in learning outcomes? (Section I), and What explains these results? (Section II). Results of the Afghanistan SABER Service Delivery Survey | 151 Box 6.1: Sampling Design for CBE schools CBE schools in Afghanistan are highly heterogeneous in terms of costs and delivery modality. There are two types of CBE schools: (i) on-budget CBE schools, which means that they are managed by the Afghan Ministry of Education (MoE) through the Provincial Education Departments (PEDs), and (ii) off-budget CBE schools, which are supervised/managed by NGOs. The CBE schools covered in the Afghanistan SABER SD survey come from the on-budget CBE schools, which are financed under the Global Partnership for Education (GPE) Project. The sample frame for these non-public-operated schools came from disparate sources and could not be assembled until six months after the sample frame for the public schools. From a final list of CBE schools run by NGOs in the relevant provinces (Ghor, Paktia and Khost), 30 CBE schools were sampled in numbers proportional to the number of CBE schools in each sampled province. A few weeks after drawing the CBE sample, it was revealed that only one in every six CBE schools in Khost province was sufficiently safe for the field teams to visit. This meant re-drawing the CBE sample in Khost among the small minority of schools that were safe enough to visit. Since we were unable to obtain the sample frame from the universe of CBE schools (on and off-budget), which would allow the proper random selection and representativeness, it is important to acknowledge that the analysis results from this report cannot be generalized since they are not representative of all CBE schools, but only of on-budget CBE schools. From the 30 CBE schools surveyed in the Afghanistan SABER SD: 6 are located in the provinces of Ghor, 13 are located in the province of Paktia and 11 are located in the province of Khost. These provinces are marked in blue color in Figure 6.1. Figure 6.1: Provinces covered in the Afghanistan SABER SD by public/CBE schools Ghor 6 CBE schools Paktia 13 CBE schools Khost 11 CBE schools NOT ANALYZED PROV W/O CBE SCHOOLS PROV W/ CBE SCHOOLS Source: SABER Service Delivery - Afghanistan 2017 152 | The Learning Crisis in Afghanistan SECTION I. HOW DO LEARNING OUTCOMES DIFFER FOR CBE SCHOOL STUDENTS AND PUBLIC SCHOOL STUDENTS? The sample of 30 CBE schools covered in the Afghanistan SABER SD survey come from the on-budget CBE schools, which are financed under the Global Partnership for Education (GPE) Project. It is important to acknowledge that the analysis results from this report cannot be generalized since they are not representative of the universe of all CBE schools, but they are representative of on-budget CBE schools. Student Results in the Language Assessment In Language, students in CBE schools tend to significantly outperform those in public schools, not only in basic Grade 1 level tasks but also in harder level Grade 4 questions involving the correct use of grammar, tenses, and reading comprehension. Figure 6.2 compares the Language assessment results between public school and CBE school students by specific items of knowledge in this subject. The figure on the left-hand side compares all the public school students to those in the CBE school, while the figure on the right-hand side compares the public rural school students to the CBE school students. The reason behind this latter comparison is based on the idea that public rural schools might be a better comparison for the CBE schools given that CBE schools are generally located in more remote areas. The Language learning gap between public school students in rural areas and CBE students is slightly larger, especially for the harder level Grade 4 tasks involving the correct use of grammar, tenses, and reading comprehension (Figure 6.2). When comparing all the public school students to the CBE students, at the lowest end of the difficulty level, 35% of Grade 4 public school students were unable to solve the most basic Grade 1 level question of identifying a picture from a given word. This number is significantly lower for CBE school students given that only 17% of students in CBE schools were unable to solve this Grade 1 level task. On the easier end of Grade 4 level questions which includes vocabulary questions, public school students perform similarly to CBE school students. Most importantly, CBE school students significantly outperform public school students in Grade 4 harder level questions involving the correct use of grammar, tenses, and reading comprehension. For instance, a higher proportion of students in CBE schools, relative to students in public schools, managed to correctly solve the complete reading comprehension exercise (25% vs 16%). Results of the Afghanistan SABER Service Delivery Survey | 153 Figure 6.2: Students' learning performance in Language by public/CBE school – Grade 4 Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 154 | The Learning Crisis in Afghanistan Figure 6.3: Students' learning performance in Language by public/CBE school and gender – Grade 4 Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 155 In Language, girls in public rural schools do not perform very differently compared to girls in CBE schools. However, boys in CBE schools significantly outperform boys in public rural schools in almost every Grade 1 and Grade 4 task, ranging from easy Grade 1 questions to harder Grade 4 questions (Figure 6.3). Student Results in the Mathematics Assessment In Mathematics, students in CBE schools also significantly outperformed those in public schools. The biggest difference is attributed to harder level questions like word problems, identifying shapes, understanding fractions and division, and solving double digit subtraction and multiplication problems. Figure 6.4 compares the Mathematics assessment results between public school students and CBE school students by specific items of knowledge in this subject. The figure on the left-hand side compares all the public school students to those in the CBE school, while the figure on the right-hand side compares the public rural school students to the CBE school students. Afghan students in both types of schools perform quite similarly in Mathematics tasks at the lowest end of the difficulty level, such as single and double digit operations. However, students in CBE schools significantly outperform students in public schools in solving questions with a higher level of difficulty such as word problems, multiplication problems involving monetary units, area calculation, etc. This difference in learning between both types of schools can range from 15% to 25%. For instance, students in CBE schools are 25% more likely to solve a multiplication problem involving monetary units compared to students in public schools. The learning gap in Mathematics between public school students in rural areas and CBE students is very similar to the one which includes all the public school students. Moreover, the learning gap in mastering Grade 4 Mathematics tasks between public and CBE school students is not only very large and but still larger than the one in Language. 156 | The Learning Crisis in Afghanistan Figure 6.4: Students' learning performance in Mathematics by public/CBE schools – Grade 4 Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 157 Figure 6.5: Student learning performance in Mathematics by public/CBE schools and gender – Grade 4 Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 158 | The Learning Crisis in Afghanistan In Mathematics, girls in CBE schools perform much better than those in public rural schools in Afghanistan, especially in tasks at the highest end of the difficulty level. Similarly, boys in CBE schools outperform significantly those in public rural schools. As suggested earlier, the biggest difference is found in harder level questions like word problems, identifying shapes, understanding fractions and division, and also double digit subtraction and multiplication (Figure 6.5). Curriculum-Adjusted Years of Schooling (CBE Schools vs Public Schools) In this section, we compare the curriculum-adjusted years of schooling between students in public schools and students in CBE schools, which is represented in Figure 6.6. CBE Afghan students have significantly more curriculum-adjusted years of schooling compared to public school students, both in Language and Mathematics. Despite this good news, the overall learning performance for CBE schools is also worrisome as more than half of Grade 4 CBE students, in both Language and Mathematics, are still performing significantly below their current grade level. In Language, CBE Afghan students are on average 0.44 curriculum-adjusted years of schooling ahead of public school students. The average student performance of Grade 4 Afghan students in CBE schools is around 1.32 curriculum-adjusted years of schooling. Even though Grade 4 CBE school students in Afghanistan also display the knowledge level of a Grade 1 student, the student performance is significantly higher compared to public schools, which only have 1.11 curriculum-adjusted years of schooling. In Mathematics, the difference in curriculum-adjusted years of schooling between public and CBE Afghan students can be as great as putting the CBE school students half year (0.5) of learning ahead of non-CBE students. Similar to the Language results, even though CBE school students are also performing significantly below their current grade level, they have 1.34 curriculum adjusted years of schooling, which is significantly higher compared to student performance in public schools. Figure 6.6: Curriculum-adjusted years of schooling by public/CBE schools – Grade 4 Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 159 Next, we compare the distribution of the curriculum-adjusted years of schooling between public and CBE school students. Figure 6.7 and Figure 6.8 present the distribution of student curriculum-adjusted years of schooling in Language and Mathematics, respectively. Figure 6.7: Distribution of curriculum-adjusted years of schooling in Language by public/CBE schools – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 Even though the results in Language suggest that CBE school students are performing significantly below their current grade level, a significantly higher proportion of students in CBE schools compared to public schools are able to perform Language questions pertaining to Grade 4 (Figure 6.7). In Language, the percentage of students that were unable to identify a picture from a word is significantly lower for CBE school students. Only 17% of CBE students belong to Grade 0 in Language as opposed to 35% of students in public schools. As a flip side to this, a slightly higher percentage of students in CBE schools (57% compared to 49% in public schools) have fully mastered the Language curriculum for Grade 1. In terms of Grade 4 level questions, the most important result is that CBE school students are significantly more able than public school students to perform advanced Grade 4 level tasks. On average, around 12% of Afghan CBE school students were able to master advanced level Grade 4 questions compared to just 2% in public schools. 160 | The Learning Crisis in Afghanistan Figure 6.8: Distribution of curriculum-adjusted years of schooling in Mathematics by public/CBE schools – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 Despite the fact that results in Mathematics are also worrisome for CBE school students in general —the average Grade 4 student still displays the knowledge level of a Grade 1 student- , evidence indicates that a higher proportion of CBE school students compared to public school students are able to master Mathematics tasks that belong to the Grade 4 curriculum or higher level (Figure 6.8). In Mathematics, the proportion of Grade 4 students who were unable to solve the most basic Mathematics tasks (i.e. single digit addition and subtraction) is lower in CBE schools relative to public schools. Still, almost half of them were not able to master the Grade 1 curriculum in Mathematics. There are fewer Grade 4 CBE school students with the Mathematics knowledge of a Grade 1 and 2 are fewer compared to in public schools. Moreover, CBE school students are more likely than public school students to perform better in Mathematics tasks that belong to the Grade 4 curriculum or higher level. In particular, 6% of CBE school students classified as Grade 4 level as stipulated by Afghanistan’s own curriculum requirements. were able to solve the current grade level tasks compared to just 2% for the public school students. Comparing Afghanistan’s CBE schools internationally, it can be inferred that they are not to be considered as a stellar alternative to public schools. Despite the fact that they perform better than public schools, their overall performance is still troublesome and needs further improvement (See Figure 6.9) Results of the Afghanistan SABER Service Delivery Survey | 161 Figure 6.9: Comparing Afghanistan’s CBE schools internationally Source: SABER Service Delivery - Afghanistan 2017, SDI, TIMSS and Pre-PIRLS 162 | The Learning Crisis in Afghanistan Putting the Public and CBE Schools Comparison in Context In order to understand what explains the learning gaps in the Language and Mathematics assessments in Afghanistan, we analyze the total test scores (i.e. percentage of total correct answers) by gender, school type, and socioeconomic background and then compare the differences across the different dimensions. In Language, the public-CBE learning gap and the gender segregated-coeducational learning gap are larger compared to the ones across students from different socioeconomic backgrounds and urban/rural schools. CBE schools have statistically significant better results in Language compared to public schools. Similarly, same sex schools outperform co-educational schools (at the classroom and school level). These results could imply that this learning gap reflects the ability of CBE schools’ and same-sex schools’ administrators and teachers to adapt better to the specific circumstances of their school (Figure 6.10). Figure 6.10: The learning gap in Language by gender, school type and socioeconomic backgrounds Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 In Mathematics, the same sex-coeducational learning gap is the largest compared to the ones across students from different socioeconomic backgrounds and other types of schools. Similar to the Language results, students in same-sex schools outperform those in co-educational schools in Mathematics. In addition to this, there is also a considerable learning gap in Mathematics between public schools and CBE schools (Figure 6.11) Results of the Afghanistan SABER Service Delivery Survey | 163 Figure 6.11 : The learning gap in Mathematics by gender, school type and socioeconomic backgrounds Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 SECTION II. WHAT EXPLAINS THESE RESULTS? WHY CBE SCHOOLS OUTPERFORM PUBLIC SCHOOLS? Section I described how CBE school students outperform students in public schools. This section goes through the critical factors that affect learning to understand whether these differed in CBE schools. We focus on three groups of variables. First, we study the difference between public school and CBE school teacher characteristics, inputs, and student characteristics. Second, we measure the difference in teacher knowledge, pedagogical knowledge, and skills in the classroom. Finally, we measure the difference in teachers’ support system. Are CBE Schools Different? Examining Teachers, Inputs, and Student Characteristics Table 6.1 compares teachers, school inputs, and student characteristics by public and CBE schools. CBE schools are very different from public schools. They only have one teacher per school in comparison to 35 teachers per schools in the average public school 30. While teachers on average are 36 years old on public schools, they are only 28 years old in CBE schools. All CBE teachers in the sample are male with little experience (an average of 3 years) compared with the public school teachers (average of 12 years). Both CBE and public school teachers have on average 14 years of education, but CBE school teachers are significant less likely to have completed a pre-service teacher training program (61% vs 83%). Another important difference is the average teacher salary. While Public school teachers get about 100 USD with a dispersion from 10 to 300 USD, CBE school teachers received about 140 USD on average. Interestingly, the there is little variation in CBE school teacher pay, with the difference between the highest and lowest CBE teacher salary being only 7 USD. 30 In the CBE School sample there is 18 Co-Ed Classroom Schools, 10 Girls Schools and 2 Boys Schools. 164 | The Learning Crisis in Afghanistan While CBE school teachers receive higher salaries, they also experience more frequent salary delays, with more than 7 out of 10 teachers reporting to have had at least one salary delay during the last school year. Finally, teachers are statistically identical on the degree to which they maximize the learning time during the lesson, with both set of teachers spending roughly 80% of the class teaching 31. Regarding school inputs, CBE schools have significantly worse school infrastructure and equipment, with only 13% of the schools having functioning toilets (versus 46%), 38% having electricity (versus 80%) and only 14% of the students having the textbooks (versus 79%). CBE schools have smaller class sizes than public schools (27 vs 44 students). Finally, students in CBE Schools were younger on average (10 vs 11-year-old) and less likely to be absent from school (13% vs 23%). Are CBE Teachers Different? Examining Teacher Content Knowledge in Language and Mathematics Figure 6.12 offers a breakdown of specific tasks on the language tests. First, we observed no difference in teacher knowledge in either the grammar or the composition task. Second, teachers CBE school outperform public school teachers in the reading comprehension task, which is consistent with the results presented above where CBE students outperform public school students in the reading comprehension task (72 vs 44). Finally, public school teachers were significantly better than CBE School teacher in the Cloze Task (44% vs 32%). In Mathematics, both set of teachers performed similarly in the items displayed in Figure 6.12. However, when computing the curriculum adjusted years of schooling, public school teachers outperformed CBE school teachers with 3.1 years of schooling for public school teachers versus 2.4 for CBE school teachers. These results appear to suggest that it is not the teacher content knowledge that explains the learning gaps between students in public schools and CBE schools. 31 We cannot compare teacher absence as in CBE School we could not follow the same procedure to compute absence as we did in Public schools. The reason being that CBE Schools are located in places where it was not possible for observers to wait for a couple of days to go back to the school to check for teachers’ absence. Results of the Afghanistan SABER Service Delivery Survey | 165 Table 6.1: Teachers, school inputs and students: Public schools versus CBE Schools Indicator Public School CBE Schools Difference Panel A – Teacher Characteristics Number of teachers per school 35 1 34*** Age 36 28 8*** Male (%) 43 100 -57*** Years of Education 14 14 0 Years of Experience 12 3 9*** Hours per day teaching 3.80 3.67 0 Monthly salary (in thousands) 7.57 10.17 -3*** Salary delays (%) 36 74 -38*** Completed training (%) 83 61 22** Time on Task (%) 80 77 3 N 1454 30 Panel B – School Inputs Observed Students-Teachers ratio 44 27 17*** Infrastructure Availability (%) 35 13 22*** Functioning toilets (%) 46 13 32*** Working Electricity (%) 80 38 42*** Multi-grade classrooms (%) 7 0 7*** Share of students without proper seats and desks (%) 52 37 15 Access to safe drinking water (%) 31 17 14 Has a boundary wall (%) 21 17 4 Temporary Classrooms (%) 15 15 0 Has at least 1 computer for student use (%) 29 - 29 Share of students wearing uniforms (%) 79 63 16** Share of students with textbooks (%) 79 14 65*** Time taken to get to school by students (mins) 17 17 0 N 170 30 Panel C – Students Characteristics Age 11 10 1** Student absence rate (%) 23 13 11*** Male (%) 50 41 9 Students off-task (%) 3 4 -2 N 3744 672 Note: * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 166 | The Learning Crisis in Afghanistan Figure 6.12: Teacher content knowledge: Public schools versus CBE schools Note: The items marked in orange show a statistically significant difference in performance. * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Are CBE Teachers Different? Examining Teacher Pedagogic Knowledge and Skills in the Classroom CBE teacher pedagogic knowledge cannot explain the differences in student performance, as CBE teachers are as good (or bad) as public school teachers (Table 6.2 Panel A). CBE teachers outperformed public school teachers in two areas of assessing and monitoring students: giving feedback on strengths and weaknesses in students’ writing using a marking scheme and monitoring and commenting on the learning progression of students (Table 6.2 Panel B and C). CBE teachers also show a significant difference on the skills they display in the classrooms compared to public school teachers (Table 6.2 Panel D). 90% of CBE teachers explained the topic of the lesson at the start and summarized what was learned at the end (versus 54%), and only around 5% of the lesson seemed unplanned to the observers (versus 15%). During their lessons, many more CBE teachers asked questions that required students to recall information, practice what was learned, apply what was Results of the Afghanistan SABER Service Delivery Survey | 167 learned to different contexts, be creative, or other questions that required higher order skills. Overall, 67% of CBE teachers mixed lower and higher order questions in their class (versus 33% public school teachers). In response to students’ answers, around 40% of CBE teachers consistently gave positive feedback and corrected mistakes without scolding students (versus 24% of public school teachers). Overall, 3 in 10 CBE teachers (versus less than 2 in 10 Public school teachers) apply the full set of beneficial skills to promote learning—structuring, planning, asking questions, creating a positive environment, and providing constructive feedback—in their lessons. Table 6.2: Comparison of teachers’ performance on pedagogic knowledge and skills in the classroom Public Schools CBE Schools Difference Panel A: Pedagogical knowledge Factual text comprehension (0-100) 40 43 -3 Formulate aims and learning outcomes (0-100) 29 29 0 Panel B: Assessing students Formulate questions to check understanding (0-100) 35 27 9 Formulate questions to apply to other contexts (0-100) 29 28 1 Assessing students’ abilities (0-100) 19 25 -6* Evaluating students’ progress (0-100) 20 46 -26*** Panel C: Monitoring System and Knowledge of Student Performance Keep record of students’ performance (%) 29 27 3 Student Performance – Single Digit Addition (%) 37 30 7 Student Performance – Single Digit Multiplication (%) 17 10 7 Student Performance – Double Digit Multiplication (%) 11 23 -12 Student Performance – Identify Letters (%) 9 13 -4 Student Performance – Read Words (%) 10 27 -17 Student Performance – Reading Comprehension (%) 9 7 3 Panel D: Skills and practices in the classroom Introduce and summarize topic of the lesson (%) 54 90 -36*** Lesson appears planned to enumerator* (%) 85 93 -8*** Ask a mix of lower and higher order questions (%) 33 67 -34*** Positive environment and feedback (%) 24 40 -16 Engages in all of the above practices (%) 14 30 -16* Note: * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 168 | The Learning Crisis in Afghanistan Are CBE Teacher Support Systems Different? To measure teachers’ support system, we focus on four areas: (i) teacher autonomy, (ii) availability of teaching materials, (iii) professional development, (iv) guidance and support on a day to day basis at the school. Table 6.3: Strengths and weakness of the teachers’ support system Indicator Public Schools CBE Schools Difference Panel A – Teaching Autonomy Curriculum (%) 71 60 11 Teaching methods (%) 82 53 28** Textbook (%) 66 63 3 Grade/Subject to teach (%) 42 47 -4 Panel B – Availability of Teaching Materials Curriculum (observed) (%) 50 47 3 Lesson plan (observed) (%) 73 80 -7 Guide on how to assess students (observed) (%) 8 50 -42*** Student Profiles (%) 30 70 -40*** Textbooks for classroom activities (%) 56 20 36*** Rubrics to evaluate for student work (%) 6 10 -4 Item banks for quizzes or examinations (%) 5 0 5** Panel C – Professional Development Participated in Professional Development (%) 44 30 14 Average Number of Days 79 37 42* Main Topic of the Training: General Pedagogical Skills (%) 61 38 23 Main Topic of the Training: Content Knowledge (%) 22 63 -41*** Main Topic of the Training: Specific Pedagogical Skills (%) 11 0 11*** Training or Follow up in the classroom (%) None 24 63 -39*** Less than 24% 61 38 23* Between 25%-50% 11 0 11*** Between 51%-75% 2 0 2*** Between 76%-100% 0 0 0 Panel D –Coaching Most important responsibility: Being on time to school (%) 86 97 -11*** Most important responsibility: Improve students score (%) 1 0 1*** Guidance to develop their lesson plan (%) 63 52 11 Evaluated by the inspector/district officer (%) 86 90 -4 Evaluates by the inspector/district officer on classroom practices (%) 59 33 26*** Note: * significant at 10%; ** significant at 5%; *** significant at 1%. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 169 As reported in Table 6.3 Panel A, CBE teachers have less autonomy to decide on their teaching methods than public school teachers. This is consistent with the finding above that more CBE teachers practice specific teaching methods in their classrooms. Panel B shows that there is no statistically significant difference in observing the curriculum or lesson plan for the day, but CBE school teachers are more likely to know the competencies students need to learn in each grade and have access to a guide on how to assess students. Public school teachers, on the other hand, are more likely to have textbooks with classrooms exercises and items banks. Professional development for CBE teachers relies more heavily (63% vs 22%) on content knowledge while for public school teachers trainings focus mostly on general pedagogical skills. Finally, Panel D of Table 6.3 shows that CBE teachers, as public school teachers, perceived that being on time to school is their primary responsibility. Similarly to public school teachers, inspectors/district officers did not evaluate CBE school teachers on what they do in the classroom but rather on other areas such as attendance. In summary, while we cannot know for certain whether the better skills teachers display in the classroom are behind the relative better performance of CBE students, these findings provide the rationale for more research on this topic. 170 | The Learning Crisis in Afghanistan References [1] Andrabi, Tahir; Das, Jishnu; Khwaja, Asim Ijaz; Vishwanath, Tara; Zajonc, Tristan. 2008. Pakistan - Learning and Educational Achievements in Punjab Schools (LEAPS) : insights to inform the education policy debate (English). Washington, DC: World Bank. [2] ASER. 2013. Annual Status of Education Report 2013. ASER Center. New Delhi. Araujo, M. Caridad, Pedro Carneiro, Yyannú Cruz-Aguayo, and Norbert Schady. 2016. [3] Bloom, Nicholas, Renata Lemos, Raffaella Sadun, and John Van Reenen. 2015. “Does Management Matter in Schools?” Economic Journal 125 (584): 647–74. [4] Bold, Tessa, Deon Filmer, Gayle Martin, Ezequiel Molina, Christophe Rockmore, Brian Stacy, Jakob Svensson and Waly Wane. 2017a. “What Do Teachers Know and Do? Does it Matter? Evidence from Primary Schools in Africa.” Policy Research Working Paper No. [5] Bold, Tessa, Deon Filmer, Ezequiel Molina, and Jakob Svensson. 2017b. “The Lost Human Capital: Teacher Knowledge and Student Achievement in Africa”, Working Paper, IIES. [6] Bold, Tessa, Bernard Gauthier, Jakob Svensson, and Waly Wane. 2010. “Delivering Service Indicators in Education and Health in Africa: A Proposal.” Policy Research Working Paper No. 5327, The World Bank. [7] Bold, Tessa, Bernard Gauthier, Jakob Svensson, and Waly Wane. 2011. “Service Delivery Indicators: Pilot in Education and Health Care in Africa.” CMI report no. 2011:8, Chr. [8] Boyd, Donald, Pamela Grossman, Hamilton Lankford, Susanna Loeb, and James Wyckoff. 2009. “Teacher Preparation and Student Achievement.” Educational Evaluation and Policy Analysis. 31:4, pp. 416-440. [9] Bruns, Barbara, Soledad De Gregorio, and Sandy Taut. 2016. “Measures of Effective Teaching in Developing Countries. RISE Working Paper No. 16/009. [10] Bruns, Barbara and Javier Luque. 2014. Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. Washington, DC: The World Bank. [11] Bruns, Barbara, Alain Mingat and Ramahatra Rakotomalala. 2003. Achieving Universal Primary Education by 2015: A Chance for Every Child. World Bank Publications, The World Bank. [12] Chaudhury, Nazmul, Jeffrey Hammer, Michael Kremer, Karthik Muralidharan and F. Halsey Rogers. 2006. “Missing in action: teacher and health worker absence in developing countries.” The Journal of Economic Perspectives 20:1, pp. 91–116. [13] Coe, Robert, Cesare Aloisi, Steve Higgins and Lee Elliot. 2014. What makes great teaching? Review of the underpinning research. The Sutton Trust, London. [14] Evans, David K. and Anna Popova. 2016. “What Really Works to Improve Learning in Developing Countries: An Analysis of Divergent Findings in Systematic Reviews”. World Bank Research Observer. 31:2, pp. 242-270. [15] Glewwe, Paul and Karthik Muralidharan. 2015. “Improving School Education Outcomes in Developing Countries: Evidence, Knowledge Gaps, and Policy Implications.” RISE Working Paper No. 15/001. [16] Hungi, Njora, Demus Makuwa, Kenneth N. Ross, Mioko Saito, Stéphanie Dolata, Frank Cappelle, Jacqueline Vellien. 2010. “SACMEQIII Project Result: Student Achievement Levels in Reading and Mathematics”. Working Document Number 1. Paris: SACMEQ. [17] Jones, S. E., Axelrad, R., & Wattigney, W. A. (2007). Healthy and safe school environment, part II, physical school environment: Results [18] Johnson, David. 2006. “Investing in Teacher Effectiveness to Improve Educational Quality in Developing Countries: Does in-service education for primary Mathematics teachers in Sri Lanka make a difference to teaching and learning?” Research in Comparative and International Education. 1:1, pp. 73-87. Results of the Afghanistan SABER Service Delivery Survey | 171 [19] Johnson, David, Andrew Cunningham and Rachel Dowling (2012), Teaching Standards and Curriculum Review, mimeo, The World Bank. [20] Kane, Thomas J. and Douglas O. Staiger. 2010. “Gathering Feedback for Teaching: Combining High-Quality Observations with Student Surveys and Achievement Gains”. MET Project Research paper. Bill & Melinda Gates Foundation. [21] Ko, James and Sammons Pamela. 2013. Effective teaching: a review of research and evidence. CfBT Education Trust. [22] Kremer, Michael, Conner Brannen, Rachel Glennerster. 2013. “The Challenge of Education and Learning in the Developing World.” Science. 340: 6130, pp. 297-300. [23] Lahire,Nathalie. 2018. Afghanistan: promoting education during times of increased fragility. Washington, D.C.: World Bank Group. [24] Lloyd, C. B., Mete, C., & Sathar, Z. A. (2005). The effect of gender differences in primary school access, type, and quality on the decision to enroll in rural Pakistan. Economic Development and Cultural Change, 53(3), 685-710. [25] M. Caridad Araujo, Pedro Carneiro, Yyannú Cruz-Aguayo, Norbert Schady; Teacher Quality and Learning Outcomes in Kindergarten , The Quarterly Journal of Economics, Volume 131, Issue 3, 1 August 2016, Pages 1415–1453, https://doi.org/10.1093/qje/qjw016 [26] Makuwa, Demus. 2011. “Characteristics of Grade 6 Teachers.” SACMEQ III, Working Paper No. 2. SACMEQ, Paris. [27] Muijs, Daniel, Leonidas Kyriakides, Greetje van der Werf, Bert Creemers, Helen Timperley and Lorna Earl. 2014. “State of the art – teacher effectiveness and professional learning.” School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice. 25:2, pp. 231-256. [28] Michelsen Institute, Bergen, Norway. [29] Muralidharan, Karthik and Venkatesh Sundararaman. 2011. “Teacher Performance Pay: Experimental Evidence from India.” Journal of Political Economy. 119:1, pp. 39-77. [30] Murnane, Richard J. and Alejandro J. Ganimian. 2014. “Improving Educational Outcomes in Developing Countries: Lessons from Rigorous Evaluations.” NBER Working Paper No. 20284. [31] Nordstrum, Lee. 2015. “Effective Teaching and Education Policy in Sub-Saharan Africa: A Conceptual Study of Effective Teaching and Review of Educational Policies in 11 Sub- Saharan African Countries.” Report, USAID. [32] OECD. 2011. Education at a Glance 2011: OECD Indicators, OECD Publishing, Paris. OECD. 2015. Education at a Glance 2015: OECD Indicators. OECD Publishing, Paris. [33] OECD, 2014."Who Are the School Truants?," PISA in Focus 35, OECD Publishing. Last available June 20th at: http://www.oecd.org/pisa/pisaproducts/pisainfocus/PISA-in-Focus-n35-(eng)-FINAL.pdf [34] OECD. (2014). PISA in Focus. (35). Retrieved from http://www.oecd.org/pisa/pisaproducts/pisainfocus/PISA- in-Focus-n35-(eng)-FINAL.pdf [35] PASEC. 2015. PASEC 2014: Education system Performance in Francophone Sub-Saharan Africa. PASEC, Dakar, Senegal. [36] Pianta, Robert, Karen LaParo, and Bridget Hamre. 2007. Classroom Assessment Scoring System—CLASS. Baltimore: Brookes. [37] Molina, Ezequiel, Scanlon, Cole, Tracy Wilichowski,. 2017. “A Global Perspective on Teacher Policies: Insights from Twenty-eight Education Systems.” Working Paper, The World Bank. [38] Stallings, Jane. 1980. “Allocated Academic Learning Time Revisited, or Beyond Time on Task.” Educational Researcher. 9:11, pp. 11-16. [39] Strong, M., Gargani, J. and Hacifazlioğlu, Ö., 2011. “Do we know a successful teacher when we see one? Experiments in the identification of effective teachers”. Journal of Teacher Education, 62(4), pp.367-382. 172 | The Learning Crisis in Afghanistan [40] Sutton, M. (1998). Girls Educational Access and Attainment. Women in the Third World: And Encyclopedia of contemporary issues, 381-396. [41] Tyler, John H., Eric S. Taylor, Thomas J. Kane, Amy L. Wooten. 2010, “Using Student Performance Data to Identify Effective Classroom Practices.” American Economic Review: Papers and Proceedings. 100:2, pp. 256- 260. [42] TIMSS & PIRLS. (2011). Pre-PIRLS 2011 International Results in Reading. TIMSS & PIRLS International Study Center. [43] UNESCO. 2013. The Global Learning Crisis: Why Every Child Deserves a Quality Education. UNESCO. Paris. [44] UNICEF. 2014. Generation 2030/Africa. UNICEF. New York. [45] Vieluf, Svenja, David Kaplan, Eckhard Klieme, Sonja Bayer. 2012. Teaching Practices and Pedagogical Innovations: Evidence from TALIS. OECD Publishing. [46] World Bank. 2015. Mozambique - Service delivery indicators : Mozambique service delivery indicators (English). Mozambique service delivery indicators brief; education. Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/287341468181503193/Mozambique-service-delivery- indicators [47] World Bank. 2018. World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. doi:10.1596/978-1-4648-1096-1. License: Creative Commons Attribution CC BY 3.0 IGO Results of the Afghanistan SABER Service Delivery Survey | 173 Appendix APPENDIX A.1: AFGHANISTAN SABER SD SAMPLING METHODOLOGY This summary is based upon data documentation by Rohit Chhabra and Owen Ozier, and upon several rounds of meetings and email exchanges in 2016 and 2017 among Ezequiel Molina, Anahita Hosseini Matin, Rohit Chhabra, and Owen Ozier. Introduction Sampling for Afghanistan began in late 2016 and concluded in mid-2017. The plan called for sampling 200 schools. Sampling in Afghanistan had several special features: high logistical costs, CBE schools, fluid security concerns, and a very specific type of gendered schools. 1. We concluded early in the process that we would first sample a subset of provinces to be visited, since visiting each province brought its own logistical costs that made little sense to incur for a small number of schools. 2. The second, CBE schools, meant that a fraction of the sample frame dealt with schools run by NGOs. These schools were disproportionately important to SABER-SD (in relation to their actual numbers), so we reserved a portion of the sample for schools of this type. However, the sample frame for these non-public-operated schools came from disparate sources and could not be assembled until six months after the sample frame for the public schools. 3. The third, fluid security concerns, had three implications. One implication was that though we had a list of public schools covering just over 1.1 million students in 34 provinces, the SABER SD team reduced that list to a set of schools that still included all 34 provinces but covered just under 900,000 students. This smaller set was the one we could sample from. The next implication was that we did not know at the time of sampling which schools would be in areas safe enough to visit at the actual survey time. This ultimately meant re-sampling CBE schools in one of the provinces, Khost, where security proved much more restrictive than had initially been thought. The last implication is that the resulting sample is representative, to the extent possible, of the places in Afghanistan that were secure enough for the teams to visit. 4. The fourth feature of this environment, gendered schools, meant that while there were both exclusively boys schools and exclusively girls schools, there were also co-educational schools that nonetheless kept classrooms sex-segregated. This implies that for each sampled school, we had to decide in advance whether a male or female classroom would be sampled. A standard feature of the sampling problem was that we wanted to be able to produce reasonably precise statistics for both girls and boys, and for both rural and urban areas, in addition to statistics for CBE schools. Upon examination of the number of schools of each type, as well as the number of children enrolled in each type of school, we devised an allocation of the sample to school types that balanced the precision issues in each group of interest against the precision of a national statistic. In 174 | The Learning Crisis in Afghanistan addition to oversampling CBE schools, would slightly oversample girls, but the rural/urban distribution did not appear uneven enough to merit further oversampling. The tradeoffs are illustrated in the table at the end of this section, comparing characteristics of a proportional sample to the slightly different sample the SABER SD team chose to implement, weighing these objectives against one another. Sampling For public schools, we had estimates of Grade 4 enrollments by gender. For CBE schools, we did not have this information. We therefore aimed to sample with probability proportional to size for all public school sampling. First stage — provinces Six provinces were sampled mechanically: three because of their political importance (Kabul City and Kabul Province in the Central region; Nangarhar in the East region); and three because they represented such a disproportionate fraction of their region that a PPS (probability proportional to size) strategy would always sample them anyway (Balkh in the North; Khost in the South; and Hirat in the West). Of the remaining 28 provinces, 15 were randomly sampled via stratified PPS, with strata simply defined as regions, yielding a total of 21 sampled provinces. Second stage — public schools Within the sampled provinces, we assigned public schools to strata defined by three characteristics: region (one of five); rural/urban; and gender (male, female, or coed). The gender category was defined empirically from enrollments reported in the sample frame: if the numbers of either males or females was zero or was very small in both absolute and proportional terms, we considered the school single- sex. If in either absolute or proportional terms neither sex dominated, we considered the school co- educational. We then did stratified PPS to sample 170 public schools, along with a number of replacements in case schools had been closed or the sample frame was in some other way erroneous. Third stage — gender Within the sampled mixed (“coed”) schools, we had to decide in advance whether to sample girls or boys. We set the overall fraction of these schools in which we would sample girls (equivalently, boys) to be equal to the overall fraction of the enrollment across all these mixed sex schools that girls (equivalently, boys) comprised. We then randomized so that at each school, the probability of girls (equivalently, boys) being sampled was roughly proportional to the fraction of that school’s enrollment that girls (equivalently, boys) comprised. Fourth step — CBE schools Six months after drawing the original sample, we received a final list of CBE schools in the relevant provinces. Unlike typical public data, and specifically unlike Afghanistan’s public school sample frame, this CBE school list did not include enrollments. We thus sampled 30 of these schools in numbers proportional to the number of CBE schools in each sampled province. Results of the Afghanistan SABER Service Delivery Survey | 175 Fifth step — Revised security for CBE schools in Khost A few weeks after drawing the CBE sample, it was revealed that only one in every six CBE schools in Khost province was sufficiently safe for the field teams to visit. This meant re-drawing the CBE sample in Khost among the small minority of schools that were safe enough to visit. Table A.1.1: Tradeoffs in statistical precision Sampling Scenario: Proportionally sampled Actual with oversampling Total Sample 200 200 Sampled G.R. 110 95 Sampled G.U. 85 75 Sampled CBE 5 30 Fraction Boys 0.3 0.3 Fraction Girls 0.2 0.3 Fraction Coed 0.5 0.4 Resulting anticipated standard errors for key school-level outcomes, within each major analytical grouping: CBE 22.4% 9.1% Public Rural 4.8% 5.4% Public Urban 5.4% 5.9% Boys 6.5% 7.0% Girls 8.0% 7.0% Coed 5.1% 6.1% National 3.5% 3.9% 176 | The Learning Crisis in Afghanistan APPENDIX A.2: THE AFGHANISTAN SABER SD TEST INSTRUMENT When designing Module 5 (Student test) we carefully studied Afghanistan Grades 1-4 Pashtu, Dari and Mathematics national curriculum. The test took elements of previously administered SDI student tests in Mozambique and Tanzania, and released question items from TIMMS 2011, TIMSS 2015 and Pre- PIRLS 2011. Every question selected for the student test can be mapped to examples from Grades 1-4 textbooks. The selected questions were then translated to Dari and Pashtu and back-translated from Dari and Pashtu into English to ensure the accuracy of translation. The level of difficulty ranges from easy to medium. The team also selected some questions directly from Afghanistan language and Mathematics textbooks. Table A.2.1 shows examples of questions used in language and Mathematics student test and Afghanistan Grade 4 national curriculum. Table A.2.1: Afghan curriculum example Afghan Curriculum Dari Version Language ________ ‫ز ﮔﮭﻮاره ﺗﺎ ﮔﻮر داﻧﺶ‬ ‫ اﻟﻒ( ﻣﯿﺠﻮﯾﻢ‬ ‫ ب( ﺑﺠﻮی‬ ‫ ج( ﺑﺠﻮﯾﻨﺪ‬ Note: Copied from Page 77 of Grade 4 Dari Textbook. The question is the title of the ‫ د( ﻣﯿﺠﻮﯾﻨﺪ‬ lesson, and it is a famous quote. The team decided to use this sentence as one of the language test questions. The English translation is: “__________ knowledge from cradle to grave.  a) Seeks  b) Seek  c) Seeking  d) be seeking” ‫ﻣﺎ ________ زﺑﺎن ﻣﺎدری ﺑﺎ‬ .‫دﯾﮕﺮان راﺑﻄﮫ ﺑﺮﻗﺮار ﻣﯿﮑﻨﯿﻢ‬ ‫ب( طﻮﺳﺖ‬ ‫اﻟﻒ( ﺗﻮﺳﺖ‬ Note: Copied from Page 40 of Grade 4 Dari Textbook. The sentence is an exercise in ‫د( در‬ ‫ج( ﺗﻮﺳﻂ‬ their language textbook. The team decided to use this exercise as part of word recognition test. The English translation is: “We connect _________ our mother tongue  a) By  b) on  c) with  d) at” Results of the Afghanistan SABER Service Delivery Survey | 177 Afghan Curriculum Dari Version Mathematics ‫ﺳﮫ ھﺰارو ﺑﯿﺴﺖ و ﺳﮫ ﺑﻄﻮر ذﯾﻞ ﻧﻮﺷﺘﮫ‬ :‫ﻣﯿﺸﻮد‬ 323A.  3023 B.  Note: Similar question from Page 4, Mathematics Textbook. This question can be 30023 C.  compared with released items from TIMMS test, and upon carefully studying the 300023 D.  national curriculum our team found that the same set of skill is covered in grade 4 Mathematics textbooks. The English translation is: “Three thousand twenty three can be written as:  A. 323  B. 3,023  C. 30,023  D. 300,023” ‫ ﺣﺼﮫ ﮐﺪام ﻣﺴﺘﻄﯿﻞ ذﯾﻞ ﺳﯿﺎه ﺷﺪه؟‬1/3 Note: Similar Question from page 128 of grade 3 Math textbook. This question was chosen since it can be compared with released items from TIMSS test, and it covers previous knowledge of students from prior grade. The English translation is: Which rectangle is 1/3 shaded? The student test (Module 5) was piloted twice in 32 schools in Kabul City and a final revised version was agreed upon to be implemented during the field survey. Due to time limits and length of each shift in Afghan schools in general, a one on one test (i.e. enumerator interviewing one student at the time) was impossible. Therefore, the team decided to conduct a written test (pen and paper). To keep the overall survey cost at an efficient level, and after studying the average number of students per classroom during piloting 32 schools in Kabul City, which 178 | The Learning Crisis in Afghanistan were up to 50 students in some of the classrooms, 20 students per classroom were randomly selected to do the written test. The test consists of 4 parts: background information, oral section, language (Dari or Pashtu; depending on the region and province language of instruction), and Mathematics. Each student was given two exam booklets. The first booklet contained three sections: the background information, oral questions, and either math or language. Before students began to answer the oral or written questions, they were required to answer some background questions about their families, what assets they have at home (such as computer, TV, electricity, etc.), their parents’ level of education, who helps them at home with their homework, etc. The enumerators read these questions out loud and asked students to complete their information. Once the background information was complete, the enumerator asked students to turn to oral questions. The enumerator read from their instruction booklet, and asked students to identify three numbers, alphabet letters, words correctly by circling them. The last question asked students to circle ‘shoes’ from a set of pictures. Once these two parts were complete the enumerator instructed the students to complete the written part of the exam. In order to minimize the chance of cheating amongst students, the language and math tests were given to every other student so that no two students sitting next to each has similar exam subject to answer. The second booklet contained either language or math. The math booklet was given to students who answered language tests in the first booklet and the language booklet was given to students who completed math questions in the first booklet. Students were given 30 minutes to complete each subject (math and language). Results of the Afghanistan SABER Service Delivery Survey | 179 APPENDIX B.1: PSYCHOMETRIC PROPERTIES To assess the reliability of the results of the student and teacher assessment presented in chapters 1 and 2 we conduct two exercises. Firstly, we measure the internal consistency of the different sections of the test, and secondly, we use item response theory to measure the item characteristics for the assessment. Internal Consistency The Cronbach’s Alpha is an internal consistency estimator. It estimates how closely related a set of items are as a group. Cronbach's alpha will generally increase as the intercorrelations among items increase. Because intercorrelations among items are maximized when all items measure the same construct, Cronbach's alpha is widely believed to indirectly indicate the degree to which a set of items measure a single unidimensional latent construct. Below we present the formula where e N is equal to the number of items, ̅ is the average inter-item covariance among the items and ̅ equals the average variance. . ̅ = ̅ + ( − 1). ̅ Cronbach's alpha result are on the good to excellent range, which allow us to conclude that the items for each section are all measuring the same latent variable. Table B.1.1: Internal consistency of the student and teacher assessment � Student Assessment Language 0.92 Mathematics 0.85 Teacher Assessment Language 0.88 Mathematics 0.81 Pedagogy 0.78 Item Response Theory and Item Characteristics Curves 32 We use item response theory to compute with what precision does each item in each section of the test allow us to estimate each test taker ability or knowledge. In order to exemplify what measures we will be presenting we use a Figure from Andrabi et al (2008) to aid the comprehension. 32 This section borrows from Andrabi, Das, Ijaz Khwaja, Vishwanath, and Zajonc LEAPS (2008) study. In particular, it draws from Annex 4 on the theoretical underpinnings of item response theory and item curve characteristics. 180 | The Learning Crisis in Afghanistan Figure B.1.1: Item characteristic curve description of an item from the test Test Characteristic Curve: The probability Discrimination: The maximum with which a person of slope of the Test Characteristic a specified knowledge Curve: a measure of how well Probability of Correct Answer obtains a correct the item differentiates between answer different groups Difficulty: The level of ability at which the probability of a correct answer is 0.5 Ability/Knowledge as standard deviations from mean The probability with which a person with no knowledge would obtain a correct answer. Source: Andrabi et al (2008) Below we compute the item characteristic curve for each item in the test as well as its attributes: guessing parameter, discrimination and difficulty. Discrimination: This attribute is estimate of the ability of the item to distinguish (or `discriminate’) between individuals of differing ability/knowledge. This ability to discriminate is described by the discrimination of the item, which measures the slope of the characteristic curve at the point of inflection (equivalent to the maximum slope) - the greater the slope at this point, the greater the ability of the item to discriminate between students above or below the point at which the slope is measured. Difficulty: The attribute is an estimate of the probability that a student of a given ability will obtain a correct answer. In terms of the diagram above, increasing the difficulty of the item will shift the item- characteristic curve to the right- a student at the same level of knowledge will have a lower probability of obtaining a correct answer (technically, the difficulty of the question is the ability/knowledge required to obtain a correct answer with a 50% probability). Guessing Parameter: This attribute is an estimate of the probability with which a student with no knowledge of the item can obtain a correct response. Below we provide the item characteristics curves graphical representation as well as estimates of each attribute. The results also confirm that while some expected variation in how well each item worked, for the most part the items worked well and the results of the test are not driven by measurement problems. Results of the Afghanistan SABER Service Delivery Survey | 181 Item Characteristics Curves for the Student’s Language Assessment Figure B.1.2: Item characteristic curve for the Language Assessment Item 1 Item 2 Item 3 Item 4 Pr(vocabulary_m_p) Pr(identifypic_m) .68.7.72 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Pr(lang1_m) Pr(lang2_m) .62 .66 .64 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 ability ability ability ability Item 5 Item 6 Item 7 Item 8 Pr(grammar_m_p) 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Pr(lang3_m) 0 .2 .4 .6 .8 1 Pr(lang4_m) 0 .2 .4 .6 .8 1 Pr(lang5_m) Pr(Response=1 | Theta) -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 ability ability ability ability Item 9 Item 10 Item 11 Item 12 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Pr(lang6_m) 0 .2 .4 .6 .8 1 Pr(lang7_m) 0 .2 .4 .6 .8 1 Pr(lang8_m) Pr(lang9_m) -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 ability ability ability ability Item 13 Item 14 Item 15 Item 16 Pr(tenses_m_p) Pr(lang10_m) Pr(lang11_m) 0 .2 .4 .6 .8 1 Pr(lang12_m) 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 ability ability ability ability Pr(comprehension_m_p) Item 17 Item 18 Item 19 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 Pr(inference) 0 .2 .4 .6 .8 1 Pr(retrieve) -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 -.1 0 .1 .2 .3 ability ability ability Standardized Ability (Theta) 182 | The Learning Crisis in Afghanistan Table B.1.2: Item characteristic curves for the Mathematics Assessment Item Guessing Std. Number Discrimination Std. Error Difficulty Std. Error parameter Error Identify picture Item 1 0.280 0.088 -0.004 0.001 0.620 0.012 from word Proportion score for 4 vocabulary Item 2 6.723 0.201 -0.021 0.001 0.090 0.006 questions Vocabulary1- Item 3 5.214 0.160 0.036 0.001 0.047 0.004 "farm" Vocabulary2- Item 4 5.595 0.172 0.018 0.001 0.046 0.004 "grow" Vocabulary3- Item 5 5.602 0.169 0.018 0.001 0.052 0.004 "father" Vocabulary4-"on" Item 6 5.993 0.179 0.036 0.001 0.031 0.003 Proportion score for 5 grammar Item 7 8.881 0.275 -0.046 0.001 0.090 0.006 questions Grammar1-"with" Item 8 4.057 0.134 0.047 0.001 0.074 0.005 Grammar2- Item 9 4.850 0.146 0.042 0.001 0.058 0.004 "sheep" Grammar3- Item 10 5.786 0.169 0.036 0.001 0.038 0.003 "behavior" Grammar4- Item 11 5.044 0.161 0.058 0.001 0.033 0.003 "third" Grammar5-"they" Item 12 4.814 0.156 0.072 0.001 0.031 0.003 Proportion score for 3 tenses Item 13 6.758 0.197 -0.005 0.001 0.063 0.005 questions Tenses1-"seek" Item 14 4.913 0.153 0.042 0.001 0.050 0.004 Tenses2-"went" Item 15 5.067 0.164 0.058 0.001 0.031 0.003 Tenses3-"is Item 16 4.361 0.163 0.095 0.001 0.024 0.002 cooking" Reading Item 17 7.572 0.224 -0.021 0.001 0.071 0.005 comprehension Retrieve Item 18 6.833 0.198 0.005 0.001 0.047 0.004 information Inference Item 19 5.916 0.172 0.018 0.001 0.052 0.004 Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 183 Pr(Response=1 | Theta) Pr(q12_m) Pr(fractions_m) Pr(tripledigitmultiplication_m) Pr(doubledigitsubtraction_m) Pr(simpleaddition_m) 0 .2 .4 .6 0 .2 .4 .6 .8 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 .2 .4 .6 .8 1 -.1 -.1 -.1 -.1 -.1 0 0 0 0 0 ability ability ability ability ability Item 1 Item 5 Item 9 Item 13 Item 17 .1 .1 .1 .1 .1 .2 .2 .2 .2 .2 Pr(q14_m) Pr(identifyshape_m) Pr(simpledivision_m) Pr(tripledigitsubtraction_m) Pr(doubledigitaddition_m) 0 .2 .4 .6 0 .2 .4 .6 .8 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 184 | The Learning Crisis in Afghanistan -.1 -.1 -.1 -.1 -.1 0 0 0 0 0 ability ability ability ability ability Item 2 Item 6 Item 10 Item 14 Item 18 .1 .1 .1 .1 .1 .2 .2 .2 .2 .2 Pr(divisionprob_m) Pr(doubledigitdivision_m) Pr(simplemultiplication_m) Pr(tripledigitaddition_m) 0 .2 .4 .6 .8 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 -.1 -.1 -.1 -.1 0 0 0 0 ability ability ability ability Item 3 Item 7 Item 15 Item 11 .1 .1 .1 .1 Standardized Ability (Theta) .2 .2 .2 .2 Pr(monetaryprob_m) Pr(tripledigitdivision_m) Pr(doubledigitmultiplication_m) Pr(simplesubtraction_m) 0 .2 .4 .6 .8 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 0 .2 .4 .6 .8 1 Item Characteristics Curves for the Student’s Mathematics Assessment -.1 -.1 -.1 -.1 Figure B.1.3: Item characteristic curve for the Mathematics Assessment 0 0 0 0 ability ability ability ability Item 4 Item 8 Item 12 Item 16 .1 .1 .1 .1 .2 .2 .2 .2 Table B.1.3: Item characteristic curves for the Mathematics Assessment Item Guessing Std. Number Discrimination Std. Error Difficulty Std. Error parameter Error Single digit Item 1 7.132 0.27244 -0.080 0.00056 0.326 0.01499 addition Double digit Item 2 7.063 0.21444 -0.015 0.00071 0.069 0.00541 addition Triple digit Item 3 5.670 0.20320 0.033 0.00088 0.024 0.00241 addition Single digit Item 4 6.536 0.20243 -0.027 0.00077 0.094 0.00664 subtraction Double digit Item 5 5.251 0.19043 0.065 0.00083 0.023 0.00235 subtraction Triple digit Item 6 4.575 0.17342 0.070 0.00096 0.033 0.00304 subtraction Single digit Item 7 5.202 0.17148 0.006 0.00074 0.078 0.00554 multiplication Double digit Item 8 6.040 0.27679 0.130 0.00126 0.003 0.00054 multiplication Triple digit Item 9 7.318 0.37025 0.144 0.00114 0.001 0.00016 multiplication Single digit Item 10 6.449 0.21161 0.028 0.00080 0.024 0.00247 division Double digit Item 11 6.890 0.25687 0.080 0.00075 0.005 0.00078 division Triple digit Item 12 6.992 0.27751 0.086 0.00092 0.004 0.00061 division Understand Item 13 3.654 0.17337 0.130 0.00126 0.027 0.00277 fractions Identify a Item 14 3.471 0.15841 0.079 0.00120 0.053 0.00424 square Solve a division word Item 15 4.113 0.19764 0.096 0.00123 0.022 0.00231 problem multiplication problem with Item 16 5.399 0.28107 0.144 0.00114 0.003 0.00065 monetary units Time Item 17 3.014 0.18505 0.157 0.00162 0.031 0.00311 problem Area Item 18 3.732 0.21714 0.182 0.00146 0.012 0.00164 calculation Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 185 Item Characteristics Curves for the Teachers Language Assessment Figure B.1.4: Item characteristic curve for the Language Assessment (First 22) Item 1 Item 2 Item 3 Item 4 .95 .2 .4 .6 .8 1 0 .2.4.6.81 .2.4.6.8 1 .75 .7 .8 .9 .85 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 5 Item 6 Item 7 Item 8 0 .2.4.6.81 0 .2.4.6.81 .4 .6 .8 1 .2 .4 .6 .8 1 Pr(Response=1 | Theta) -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 9 Item 10 Item 11 Item 12 0 .2.4.6.81 0 .2.4.6.81 .2 .4 .6 .8 1 0 .2.4.6.8 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 13 Item 14 Item 15 Item 16 0 .2.4.6.81 .2 .4 .6 .8 1 0 .2 .4 .6 .8 0 .2.4.6.8 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 17 Item 18 Item 19 Item 20 0 .1.2.3.4.5 0 .2 .4 .6 .8 0 .1 .2 .3 .2.4.6.8 1 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 21 Item 22 0 .2 .4 .6 .8 0 .2.4.6.81 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Standardized Ability (Theta) 186 | The Learning Crisis in Afghanistan Figure B.1.5: Item characteristic curves for the Language Assessment (22-44) Item 23 Item 24 Item 25 Item 26 .25 .05.1.15.2.25 0 .2 .4 .6 .8 0 .2.4.6.81 .1 0.05 .2 .15 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 27 Item 28 Item 29 Item 30 0 .2 .4 .6 .8 0 .1 .2 .3 .4 0 .2.4.6.81 0 .2.4.6.81 Pr(Response=1 | Theta) -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 31 Item 32 Item 33 Item 34 0 .05 .1 .15 0 .2 .4 .6 0 .1.2.3.4.5 0 .1 .08 .06 .04 .02 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 35 Item 36 Item 37 Item 38 .04 0 .1.2.3.4.5 0 .2.4.6.81 0 .2.4.6.81 .03 .02 0.01 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 39 Item 40 Item 41 Item 42 0 .2 .4 .6 .8 0 .2.4.6.81 0 .2.4.6.81 0 .2.4.6.81 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Item 43 Item 44 0 .2.4.6.81 0 .2.4.6.8 -.2 -.1 0 .1 .2 -.2 -.1 0 .1 .2 Standardized Ability (Theta) Results of the Afghanistan SABER Service Delivery Survey | 187 Table B.1.4: Item characteristic curves for the Language Assessment Discrimination Std. Error Difficulty Std. Error Guessing Std. Error parameter Item 1 5.024 0.455 -0.106 0.001 0.305 0.046 Item 2 1.139 0.267 -0.062 0.002 0.679 0.045 Item 3 5.512 0.367 -0.050 0.001 0.100 0.019 Item 4 5.182 0.375 -0.059 0.001 0.139 0.024 Item 5 5.211 0.338 -0.029 0.001 0.074 0.014 Item 6 4.563 0.309 0.009 0.001 0.052 0.010 Item 7 4.148 0.427 -0.109 0.002 0.369 0.048 Item 8 3.623 0.332 -0.074 0.001 0.274 0.037 Item 9 6.211 0.366 -0.017 0.001 0.033 0.007 Item 10 5.891 0.363 -0.047 0.001 0.074 0.015 Item 11 3.496 0.302 -0.059 0.001 0.228 0.032 Item 12 5.776 0.437 0.065 0.001 0.006 0.002 Item 13 5.863 0.354 0.012 0.001 0.022 0.005 Item 14 3.739 0.300 -0.050 0.001 0.186 0.028 Item 15 5.090 0.365 0.052 0.001 0.014 0.004 Item 16 6.766 0.528 0.084 0.001 0.002 0.001 Item 17 4.542 0.660 0.133 0.003 0.003 0.002 Item 18 2.840 0.272 0.039 0.001 0.101 0.017 Item 19 4.327 0.312 -0.047 0.001 0.140 0.023 Item 20 2.576 0.706 0.116 0.002 0.003 0.002 Item 21 6.913 0.409 -0.040 0.001 0.043 0.010 Item 22 5.102 0.435 0.080 0.001 0.007 0.002 Item 23 0.740 0.183 0.042 0.001 0.015 0.006 Item 24 5.905 0.361 0.017 0.001 0.019 0.005 Item 25 3.418 0.384 0.091 0.002 0.025 0.007 Item 26 0.578 0.156 0.007 0.002 0.059 0.015 Item 27 5.022 0.433 0.073 0.001 0.008 0.003 Item 28 5.519 1.815 0.133 0.003 0.000 0.000 Item 29 5.132 0.337 0.024 0.001 0.027 0.006 Item 30 5.152 0.331 0.009 0.001 0.036 0.008 Item 31 4.701 0.627 0.119 0.002 0.004 0.002 Item 32 8.000 1.817 0.147 0.000 0.000 0.000 Item 33 0.282 0.060 0.007 0.002 0.003 0.002 Item 34 0.143 0.036 0.007 0.002 0.000 0.000 Item 35 6.772 0.389 -0.020 0.001 0.026 0.006 Item 36 5.171 0.790 0.136 0.003 0.002 0.001 Item 37 0.088 0.027 0.007 0.002 0.000 0.000 Item 38 6.603 0.382 -0.005 0.001 0.020 0.005 Item 39 3.904 0.395 0.084 0.001 0.018 0.005 Item 40 6.761 0.393 0.009 0.001 0.014 0.004 Item 41 5.509 0.343 0.000 0.001 0.036 0.008 Item 42 6.342 0.373 -0.005 0.001 0.024 0.006 Item 43 4.413 0.311 0.026 0.001 0.043 0.009 Item 44 3.924 0.300 0.035 0.001 0.052 0.011 188 | The Learning Crisis in Afghanistan Item Characteristics Curves for the Teacher’s Mathematics Assessment Figure B.1.6: Item characteristic curves for the Mathematics Assessment Single digit addition Single digit subtraction Double digit addition Double digit subtraction .5 .6 .7 .8 .9 1 .6 .7 .8 .9 1 0 .2 .4 .6 .8 1 .2 .4 .6 .8 1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 Triple digit addition Double digit multiplication Triple digit multiplication Double digit division 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 .2 .4 .6 .8 1 Pr(Response=1 | Theta) -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 Triple digit division Division problem Venn diagrams Problem involving time and distance 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 Simple algebraic problem Difficult math story Understand fractions Sequence 0 .2 .4 .6 .8 .2 .4 .6 .8 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 Problem involving monetary units Perimeter of a shape Area of a shape Interpret data in a graph 0 .2 .4 .6 .8 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 -.3 -.2 -.1 0 .1 Standardized Ability (Theta) Results of the Afghanistan SABER Service Delivery Survey | 189 Table B.1.5: Item characteristic curves for the Mathematics Assessment Std. Std. Guessing Std. Discrimination Error Difficulty Error parameter Error Single digit addition 5.088 0.879 -0.253 0.000 0.489 0.165 Single digit subtraction 3.103 0.806 -0.253 0.000 0.577 0.124 Double digit addition 4.464 0.421 -0.095 0.002 0.061 0.021 Double digit subtraction 2.810 0.413 -0.190 0.002 0.345 0.082 Triple digit addition 4.728 0.450 -0.099 0.002 0.058 0.021 Double digit multiplication 5.980 0.484 -0.053 0.002 0.010 0.004 Triple digit multiplication 5.484 0.421 -0.028 0.002 0.008 0.003 Double digit division 3.826 0.467 -0.139 0.003 0.169 0.052 Triple digit division 7.748 0.599 -0.055 0.001 0.002 0.001 Division problem 5.082 0.443 -0.050 0.002 0.020 0.008 Venn diagrams 4.136 0.382 -0.049 0.002 0.040 0.014 Problem involving time and distance 5.192 0.407 0.015 0.001 0.004 0.002 Simple algebraic problem 5.506 0.435 0.030 0.001 0.002 0.001 Difficult math story 6.021 0.443 -0.024 0.001 0.004 0.002 Understand fractions 5.551 0.451 -0.043 0.002 0.011 0.005 Sequence 2.462 0.308 -0.049 0.002 0.130 0.035 Solve multiplication problem involving 5.618 0.440 -0.040 0.002 0.009 0.004 monetary units Perimeter of a shape 7.502 0.521 0.015 0.001 0.000 0.000 Area of a shape 7.855 0.551 0.025 0.001 0.000 0.000 Interpret data in a graph 5.326 0.485 0.053 0.001 0.001 0.001 190 | The Learning Crisis in Afghanistan Item Characteristics Curves for the Teacher’s Pedagogy Assessment Figure B.1.7: Item characteristic curves for the Pedagogy Assessment Factual text comprehension Aims and Learning outcomes Formulating Qz to check understanding 1 1 1 .8 .8 .8 .6 .6 .6 .4 .4 Pr(Response=1 | Theta) .4 .2 .2 .2 0 0 -.1 0 .1 .2 -.1 0 .1 .2 -.1 0 .1 .2 Formulating Qz for other contexts Assessing Student Ability Assessing Student Progress 1 1 1 .9 .8 .8 .8 .6 .6 .7 .4 .4 .6 .2 .2 .5 0 0 -.1 0 .1 .2 -.1 0 .1 .2 -.1 0 .1 .2 Standardized Ability (Theta) Table B.1.6: Item characteristic curves for the Pedagogy Assessment Guessing Discrimination Std. Error Difficulty Std. Error parameter Std. Error Factual text comprehension 3.202 0.254 -0.065 0.001 0.329 0.027 Aims and Learning outcomes 6.283 0.341 -0.050 0.001 0.101 0.013 Formulating Qz to check 6.046 0.309 -0.024 0.001 0.051 0.008 understanding Formulating Qz for other contexts 8.977 0.493 0.016 0.001 0.004 0.001 Assessing Student Ability 4.484 0.282 -0.023 0.001 0.106 0.015 Assessing Student Progress 3.726 0.432 -0.111 0.001 0.470 0.041 Results of the Afghanistan SABER Service Delivery Survey | 191 APPENDIX B.2. ROBUSTNESS CHECKS FOR CURRICULUM-ADJUSTED YEARS OF SCHOOLING DEFINITIONS The section presents three robustness checks on the definition of the curriculum-adjusted years of schooling: (i) grade qualification criteria is made easier (ii) missing responses are excluded, and (ii) responses are assumed as guessed responses. Easier Criteria for Qualification on Each Grade Level We perform a robustness check on the definition of the Curriculum-adjusted years of schooling by easing up on the conditions to qualify for each grade level. The below statistics show how student’s performance changes if the criteria for qualification is made easier per grade. Figure B.2.1: Robustness check— distribution of curriculum— adjusted years of schooling in Language – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 Afghanistan Curriculum Standard SABER SD Language Test Results Grade 1: Requirements for qualifying as Grade 1 level is the same 35% of all students were unable to point to a picture of shoes as before. Students must be able to correctly identify a picture when the enumerator asked them to. Around half of the from a given word. (Criteria not changed) students could do this task but failed to do any more advanced tasks from the Grade 4 level. Grade 4: Requirements for qualifying as Grade 4 easy and The percentage of students qualifying for the easy level intermediate levels are lowered from 75% to 60%. While, for the increases from 12% to 18%. While, for the intermediate level, advanced Grade 4 level, instead of needing 2 out of 3 correct the percentage of students drops slightly. This clearly reflects responses to classify for this level, the easier threshold requires the easier threshold for the Grade 4 advanced level. The students to only solve correctly 1 out of 3 questions. percentage of students who make it to advanced Grade 4 level goes up from 2% to 6 %. 192 | The Learning Crisis in Afghanistan Figure B.2.2: Robustness Check— Distribution of curriculum- adjusted years of schooling in Mathematics – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 Afghanistan Curriculum Standard SABER SD Mathematics Test Results Grade 1: Students are expected to know how to add and subtract Student results are unchanged. single digits. (Criteria not changed) Grade 2: Students are expected to know how to do double digit Students results decrease from 28% to 16% by making OR triple digit addition, as well as all grade 1 level questions. criteria to qualify for grade 3 onwards easier. (Criteria not changed) Grade 3: Students are now expected to solve 2 out of 3 of the By allowing for any 2 of the 3 to be correct, the percentage previously required skills: multiplying single digits (7x8), dividing of students who qualify for Grade 3 increases from 4% to single digits (6/3) and subtracting double digits (57-49). 13%. Grade 4: Students are expected to additionally know how to a) The percentage of students classified as Grade 4 increases divide double or triple digits by a single digit, and b) match slightly from 2% to 3%. fraction with a shaded figure or solve a simple division problem correctly. (Criteria not changed) Grade 5: Students are now expected to solve 1 out of 3 word The percentage of students that classified for Grade 5 problems, instead of 2 out of 3 (e.g. double (37x40) or triple digit increases from 0.7% to 2.6%. multiplication (214x104), and also word problems involving multiplication with monetary units, time problem and area calculation) Which questions were most likely to be skipped? Not all students answered every question in the exam. This would be a cause for concern if students chose to skip particular questions. Table B.2.1 shows the number of skipped questions and the percentage of students who skipped them. Results of the Afghanistan SABER Service Delivery Survey | 193 Table B.2.1: Number of questions skipped and number of students who skipped them Language Mathematics Number of Number of students % of students Number of students % of students questions skipped 0 214 5.72 179 4.78 1 162 4.33 177 4.73 2 117 3.13 156 4.17 3 133 3.55 196 5.24 4 125 3.34 215 5.74 5 129 3.45 223 5.96 6 112 2.99 454 12.13 7 117 3.13 278 7.43 8 109 2.91 238 6.36 9 or more 2,526 67.47 1,628 43.48 Source: SABER Service Delivery - Afghanistan 2017 Of the 3,744 students who took the Language test, the majority (67%) skipped most of the questions; of those who took the math test 43% skipped half of the questions. Only around 5% of the students could answer nearly every question, both in Language and Mathematics. Boys are significantly more likely than girls to skip Language questions, while girls are significantly more likely than boys to skip Mathematics questions (See Tables C.5 and C.6 for more detail in Appendix C: Additional Tables). Moreover, Figure B.2.3 and Figure B.2.4 display which questions were most likely to be skipped by students in the Language and Mathematics test, respectively. In the Afghanistan SABER SD 2017, the questions in the Language and Mathematics tests were organized in increasing level of difficulty. In general, Afghan students were more likely to skip the last and most difficult questions of both tests. 194 | The Learning Crisis in Afghanistan Figure B.2.3: Questions more likely to be skipped in Language – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 In Language, 35-40% of Grade 4 Afghan students skipped the hardest questions. None of the students skipped the easiest Grade 1 level language question to identify a picture from a word. However, students skipped the Grade 4 reading comprehension and tenses questions around 40% of the time, and questions that required them to fill in the blanks with vocabulary and grammar around 32-35% of the time. Afghan students were more likely to skip Grade 4 Mathematics questions compared to Grade 4 Language questions. Word problems were the most likely to be skipped. Around 70% of students skipped word problems despite the low-level nature of the questions i.e. identifying a square despite being a concept that students in Afghanistan learn around Grade 1 or Grade 2. Similarly, 75% of the students skipped Grade 4 level division word problem, while questions that utilize the same skills of single digit subtraction and double digit division were only skipped by 18% and 47% of the students, respectively. These results pinpoint a difficulty in answering questions that require strong comprehension skills. However, it could also be the case that students ran out of time and skipped the difficult questions at the end of the exam. Results of the Afghanistan SABER Service Delivery Survey | 195 Figure B.2.4: Questions more likely to be skipped in Mathematics – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 How do test scores change when removing questions that were not answered? To rule out the theory students did not have enough time to solve the questions at the end of the exam or any item non-response bias, it is useful to analyze the curriculum-adjusted years of schooling results using only questions that were actually attempted by students. By excluding all missing responses, this analysis allows us to see if the performance is affected significantly compared to the original estimation in which non-attempted questions are zero. Figure B.2.5 shows that in Language, the percentage of students achieving easy Grade 4 status increases from 12.5% to around 20%, while advanced Grade 4 status is only achieved by 4% as opposed to 2% when using the original definition. This shows that even among questions that were answered, the patterns do not change for the language assessment. 196 | The Learning Crisis in Afghanistan Figure B.2.5: Curriculum-adjusted years of schooling in Language — original versus excluding missing values – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 After excluding the missing responses, the number of students who qualify as having Grade 2-5 level knowledge in Mathematics increases as shown in Figure B.2.6. The proportion of students that can answer Grade 4 level difficulty questions increase four-fold from 2.1% to 8.7% if we only observe questions that were attempted by students. Similarly, those that are able to reach Grade 5 status also increase from 0.9% to 3.7%. However, figures for Grade 1 drop when excluding skipped questions. Clearly, skipping questions contributes to an increase in student performance in Mathematics compared to the original estimation given that a higher proportion of students are able to master more difficult tasks, while a smaller proportion of students are unable to solve Grade 1 tasks. However, only 4 out of 16 of the Mathematics questions were multiple-choice, while the rest required students to calculate responses and write them down, thus decreasing the chance of a “lucky guess”. Figure B.2.6: Curriculum-adjusted years of schooling in Mathematics — original versus excluding missing values – Grade 4 Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 197 Learning to Guess Another method of dealing with missed responses is to assign a randomly selected response to the question skipped by a student. By generating multiple responses, a bootstrapped version of student results is generated using the average across 200 repetitions. This measure provides an estimate of how the average scores would change if students would have attempted all the questions. This estimation is done using only the multiple-choice questions in Language and Mathematics. More specifically, all the 23 Language questions are used to construct the Language score because they are all multiple-choice question types. While only 4 out of the 16 Mathematics questions used to construct the Mathematics score are multiple choice questions. Figure B.2.7 represents the average students’ performance in Language by Grade level when an answer is assigned randomly to students who didn't answer a multiple-choice question. After aggregating the items by Grade level, the results show that less Afghan students qualify for Grade 0 and 1 compared to the original estimation (i.e. they are now more able to solve easy questions). Similarly, more students qualify for the Grade 4 level, especially easy (19%) and advanced levels (3.5%). Overall, the predicted curriculum-adjusted years of schooling suggests that Afghan student performance in Language is below what randomness would imply. Figure B.2.7: Observed and predicted distribution of curriculum-adjusted years of schooling in Language – Grade 4 Note: The predicted curriculum-adjusted years of schooling in Language is calculated using the 23 Language questions of the test, which are all multiple-choice type. A bootstrap estimation of 200 repetitions is used to assign an answer randomly to students who didn’t answer a multiple-choice question. Source: SABER Service Delivery - Afghanistan 2017 Figure B.2.8 represents the average student’s performance in Mathematics by Grade level when an answer is assigned randomly to students who didn't answer a multiple-choice question. In Mathematics, the predicted curriculum-adjusted years of schooling indicate that Afghan students would have performed very similar to the original estimation if they had answered the missing questions randomly. However, in this case again it should be noted that only 4 out of 16 of the Mathematics questions were multiple-choice, while the rest required students to calculate responses and write them down, thus decreasing the chance of a “lucky guess”. 198 | The Learning Crisis in Afghanistan Overall, the predicted curriculum-adjusted years of schooling in Mathematics remains relatively unchanged compared to the original estimation, despite using this technique. Figure B.2.8: Observed and predicted distribution of curriculum-adjusted years of schooling in Mathematics – Grade 4 Note: The predicted curriculum-adjusted years of schooling in Mathematics is calculated using the 4 out of 16 Mathematics questions of the test, which are multiple-choice type. A bootstrap estimation of 200 repetitions is used to assign an answer randomly to students who didn’t answer a multiple-choice question. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 199 APPENDIX C: ADDITIONAL TABLES Table C.1: Student’s learning performance in language by type of language (Dari vs. Pashtu) – Grade 4 Diff Dari-Pashtu Total Dari Pashtu Mean diff. p-value Identify picture from word 65 57 85 -27.24 (0.005)*** Proportion score for 4 vocabulary questions 35 37 36 0.93 (0.468) Vocabulary1-"farm" 34 35 39 -4.03 (0.158) Vocabulary2-"grow" 36 39 36 3.35 (0.467) Vocabulary3-"father" 38 39 39 -0.11 (0.575) Vocabulary4-"on" 33 36 31 4.50 (0.929) Proportion score for 5 grammar questions 31 35 26 9.51 (0.56) Grammar1-"with" 33 38 27 10.23 (0.968) Grammar2-"sheep" 34 38 30 8.27 (0.55) Grammar3-"behavior" 34 39 31 8.25 (0.738) Grammar4-"third" 28 29 28 0.77 (0.836) Grammar5-"they" 25 32 12 20.04 (0.012)** Proportion score for 3 tenses questions 26 30 20 9.93 (0.238) Tenses1-"seek" 32 38 25 12.95 (0.367) Tenses2-"went" 27 33 17 15.47 (0.064)* Tenses3-"is cooking" 20 20 19 1.37 (0.67) Reading comprehension 16 18 14 3.59 (0.656) Retrieve information 17 19 14 4.77 (0.743) Inference 15 16 14 2.40 (0.59) Source: SABER Service Delivery - Afghanistan 2017 200 | The Learning Crisis in Afghanistan Table C.2: Student’s learning performance in language by urban/rural schools – Grade 4 Diff Urban Rural Mean Urb – Rur Total Schools Schools Diff p-value Identify picture from word 65 64 66 -2 (1.23) Proportion score for 4 vocabulary questions 35 38 32 6 (0.173) Vocabulary1-"farm" 34 35 32 4 (0.479) Vocabulary2-"grow" 36 40 32 8 (0.124) Vocabulary3-"father" 38 42 34 8 (0.093)* Vocabulary4-"on" 33 36 30 6 (0.226) Proportion score for 5 grammar questions 31 34 28 7 (0.063)* Grammar1-"with" 33 35 31 4 (0.316) Grammar2-"sheep" 34 37 31 6 (0.145) Grammar3-"behavior" 34 40 29 11 (0.01)*** Grammar4-"third" 28 31 25 7 (0.095)* Grammar5-"they" 25 28 22 6 (0.178) Proportion score for 3 tenses questions 26 30 23 7 (0.051)* Tenses1-"seek" 32 36 29 7 (0.118) Tenses2-"went" 27 32 23 9 (0.02)** Tenses3-"is cooking" 20 22 18 4 (0.21) Reading comprehension 16 18 14 4 (0.093)* Retrieve information 17 19 15 4 (0.112) Inference 15 17 13 4 (0.104) Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 201 Table C.3: Student’s learning performance in mathematics by urban/rural schools – Grade 4 Diff Urban Rural Urb – Rur Total Schools Schools Mean Diff p-value Single digit addition 77 78 76 3 (0.39) Double digit addition 50 49 50 -1 (1.162) Triple digit addition 27 31 23 8 (0.039)** Single digit subtraction 52 57 48 9 (0.038)** Double digit subtraction 23 26 20 6 (0.07)* Triple digit subtraction 23 30 16 14 (0)*** Single digit multiplication 41 42 40 2 (0.625) Double digit multiplication 9 9 8 1 (0.647) Triple digit multiplication 6 6 6 0 (1.16) Single digit division 31 32 30 2 (0.624) Double digit division 17 18 15 3 (0.318) Triple digit division 15 16 14 2 (0.438) Understand fractions 15 14 15 0 (1.06) Identify a square 22 24 21 3 (0.448) Solve a division word problem 15 20 11 9 (0)*** Multiplication involving monetary units 8 8 7 1 (0.538) Time problem 12 13 11 2 (0.497) Area calculation 8 6 9 -3 (1.799) Source: SABER Service Delivery - Afghanistan 2017 202 | The Learning Crisis in Afghanistan Table C.4: Language questions and percentage of skipped questions by gender – Grade 4 Percentage of students Mean diff test Diff B-C Boys Girls B-C p-value Identify picture from word 0.08 0.00 0.08 (0.154) Proportion score for 4 vocabulary questions 43.74 23.50 20.24 (0)*** Vocabulary1-"farm" 48.46 31.65 16.81 (0)*** Vocabulary2-"grow" 51.61 32.81 18.80 (0)*** Vocabulary3-"father" 53.50 34.72 18.78 (0)*** Vocabulary4-"on" 56.65 38.22 18.42 (0)*** Proportion score for 5 grammar questions 36.16 26.55 9.61 (0)*** Grammar1-"with" 49.27 41.63 7.64 (0)*** Grammar2-"sheep" 54.45 45.07 9.38 (0)*** Grammar3-"behavior" 55.10 43.43 11.67 (0)*** Grammar4-"third" 57.52 45.67 11.85 (0)*** Grammar5-"they" 59.54 46.81 12.73 (0)*** Proportion score for 3 tenses questions 43.83 37.52 6.31 (0)*** Tenses1-"seek" 51.40 44.68 6.72 (0)*** Tenses2-"went" 52.51 45.46 7.06 (0)*** Tenses3-"is cooking" 53.13 45.98 7.15 (0)*** Proportion score for a task devoted to text 41.63 36.47 5.16 (0.021)** comprehension Reading comprehension1-retrieve information 53.16 46.47 6.69 (0)*** Reading comprehension2-inference 56.43 52.01 4.42 (0.004)*** Reading comprehension3-retrieve information 57.53 51.80 5.73 (0.001)*** Reading comprehension4-retrieve information 58.39 54.95 3.44 (0.018)** Readingcomprehension5-retrieve information 60.74 57.06 3.67 (0.01)*** Reading comprehension6-inference 65.29 66.64 -1.35 (0.951) Reading comprehension8-inference 69.24 65.94 3.30 (0.027)** Reading comprehension9-retrieve information 65.10 64.68 0.43 (0.719) Reading comprehension10-inference 69.76 68.64 1.12 (0.192) Reading comprehension11-inference 70.29 69.20 1.08 (0.119) Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 203 Table C.5: Mathematics questions and percentage of skipped questions by gender – Grade 4 Percentage of students Mean diff test Diff B-C Boys Girls B-C p-value Single digit addition 8.66 7.83 0.82 (0.603) Double digit addition 13.22 11.61 1.62 (0.981) Triple digit addition 20.48 20.57 -0.10 (0.185) Single digit subtraction 18.61 16.11 2.50 (0.214) Double digit subtraction 27.63 27.58 0.04 (0.259) Triple digit subtraction 36.83 40.20 -3.37 (0.041)** Single digit multiplication 24.10 26.27 -2.17 (0.007)*** Double digit multiplication 36.79 42.81 -6.02 (0)*** Triple digit multiplication 45.70 53.20 -7.50 (0)*** Single digit division 39.66 42.00 -2.34 (0.008)*** Double digit division 43.89 51.01 -7.12 (0)*** Triple digit division 54.33 58.41 -4.08 (0)*** Understand fractions 68.29 70.22 -1.93 (0.796) Identify a square 69.55 67.82 1.74 (0.066)* Solve a division word problem 68.28 62.77 5.52 (0)*** Solve multiplication problem involving monetary units 76.17 75.13 1.04 (0.1)* Time problem 67.45 67.41 0.04 (0.579) Area calculation 73.22 75.74 -2.52 (0.48) Source: SABER Service Delivery - Afghanistan 2017 204 | The Learning Crisis in Afghanistan Table C.6: Comparison of teachers’ time use Boys Girls COED COED Urban Rural Statistically Schools Schools Schools Classroom Schools Schools Significant Overall (B) (G) (C) (Class) (U) (R) Difference School absence 10.3 11.3 8 11.4 10.4 10.3 10.3 None rate (%) Classroom 15.4 19.3 11.5 16.4 11.3 18.2 10.8 U>R(*) absence rate (%) Time on task (%) 79.4 82.1 75.8 79.9 74.7 80 77.7 None Schedule GB(**), G>Class(***), Grammar 0.75 0.71 0.79 0.76 0.63 0.76 0.71 C>Class(**),U> R(*) G>Class(**), Cloze task 0.44 0.41 0.48 0.44 0.36 0.45 0.4 C>Class(**) Composition G>Class(**), 0.29 0.27 0.32 0.3 0.22 0.3 0.28 task C>Class(**) Reading G>B(*), 0.44 0.41 0.54 0.43 0.34 0.42 0.48 Comprehension G>Class(**) Observations 1,089 257 338 408 86 656 433 Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 205 Table C.8: Teachers’ performance in mathematics by type of task Boys Girls COED COED Urban Rural Statistically Schools Schools Schools Classroom Schools Schools Significant Overall (B) (G) (C) (Class) (U) (R) Difference Single digit addition 0.99 0.99 0.98 0.99 0.99 0.99 0.98 None Single digit subtraction 0.97 0.98 0.95 0.97 0.97 0.97 0.96 None Double digit addition 0.82 0.71 0.85 0.87 0.82 0.83 0.79 G>B(*), C>B(**), Double digit subtraction 0.9 0.95 0.89 0.88 0.87 0.94 0.82 U>R(***) Triple digit addition 0.84 0.82 0.86 0.86 0.8 0.86 0.82 None G>Class(**), Double digit 0.78 0.74 0.78 0.83 0.62 0.8 0.73 C>Class(***), multiplication U>R(*) Triple digit 0.67 0.69 0.65 0.68 0.63 0.7 0.61 U>R(*) multiplication B>G(***), B>C(***), Double digit division 0.89 0.96 0.86 0.88 0.78 0.91 0.84 B>Class(***), U>R(**) Triple digit division 0.8 0.74 0.81 0.83 0.75 0.84 0.72 U>R(**) Division problem 0.76 0.75 0.74 0.8 0.68 0.77 0.74 None Venn diagrams 0.59 0.55 0.69 0.59 0.44 0.63 0.52 G>B(*), G>Class(***) Problem involving time 0.36 0.39 0.42 0.34 0.23 0.38 0.33 G>Class(**) and distance Simple algebraic B>Class(*), 0.44 0.44 0.41 0.49 0.26 0.47 0.38 problem C>Class(**) Difficult math story 0.67 0.63 0.7 0.7 0.52 0.69 0.62 G>Class(*), C>Class(*) B>Class(**), Understand fractions 0.74 0.77 0.73 0.76 0.55 0.74 0.74 G>Class(**), C>Class(**) Sequence 0.65 0.58 0.73 0.65 0.56 0.65 0.63 G>B(*), G>Class(**) G>Class(**), Multiplication with 0.73 0.66 0.76 0.79 0.56 0.77 0.65 C>Class(**), monetary units U>R(**) B>Class(**), Perimeter of a shape 0.5 0.53 0.52 0.51 0.32 0.54 0.42 G>Class(*), C>Class(*), U>R(*) G>Class(*), Area of a shape 0.45 0.47 0.49 0.43 0.32 0.51 0.33 U>R(**) Interpret data in a 0.33 0.33 0.35 0.34 0.26 0.36 0.27 None graph (Telephone Bill 799 189 246 291 73 481 318 Exercise) Observations Source: SABER Service Delivery - Afghanistan 2017 206 | The Learning Crisis in Afghanistan Table C.9: Pedagogical knowledge and skills in the classroom Boys Girls COED COED Urban Rural Statistically Schools Schools Schools Classroom Schools Schools Significant Overall (B) (G) (C) (Class) (U) (R) Difference Panel A: Pedagogical knowledge Factual text 40 41 38 39 45 37 49 R>U(***) comprehension (0-100) Formulate aims and 29 30 31 28 24 31 25 None learning outcomes (0-100) Panel B: Assessing students Formulate questions to check understanding 35 33 45 31 34 37 29 G>C(*),U>R(*), (0-100) Formulate questions to G>B(**), G>C(**), apply to other contexts 29 26 42 26 23 33 19 G>Class(**), U>R(***) (0-100) Assessing students’ 19 21 18 19 12 21 13 B>Class(**), abilities G>Class(**), U>R(***) Evaluating students’ 20 19 23 19 16 22 13 U>R(**), progress Panel C: Monitoring System and Knowledge of Student Performance Keep record of students’ G>B(***), G>C(*), 29 12 51 26 41 29 31 performance (%) Class>B(**) Single Digit Addition (%) 37 34 31 48 25 37 37 None Single Digit 17 15 23 15 16 21 14 None Multiplication (%) Double Digit G>B(**), C>B(**), 11 0 8 15 27 13 10 Multiplication (%) Class>B(**) Identify Letters (%) 9 10 2 12 12 8 10 C>G(*) Read Words (%) 10 15 8 7 11 10 9 W>CE(*) Reading 9 3 5 11 21 11 8 Class>B(*) Comprehension (%) Panel D: Skills and 0 0 0 0 0 0 0 practices in the classroom Introduce and summarize 54 55 41 58 77 54 54 Class>G(**) topic of the lesson (%) Lesson appears planned 85 86 86 87 60 91 67 B>Class(**), to enumerator* (%) G>Class(**), C>Class(**), U>R(***) Ask a mix of lower and 33 32 41 30 15 32 35 G>Class(**) higher order questions (%) Positive environment and 24 31 25 17 36 20 37 R>U(*) feedback (%) Engages in all of the 14 23 16 7 12 14 14 None above practices (%) Results of the Afghanistan SABER Service Delivery Survey | 207 Table C.10: Strengths and weakness of the teachers’ support system Boys Girls COED COED Urban Rural Statistically Schools Schools Schools Classroom Schools Schools Significant Overall (B) (G) (C) (Class) (U) (R) Difference Panel A: Teaching Autonomy Curriculum (%) 71 68 72 73 55 73 61 None C>Class(**), Teaching methods (%) 82 72 83 90 60 85 70 U>R(*) Textbook (%) 66 71 68 62 44 66 68 None B>Class(**), Grade/Subject to teach (%) 42 38 38 52 4 46 30 G>Class(***), C>Class(***) Panel B – Availability of Teaching Materials G>Class(*), Curriculum (observed) (%) 50 41 56 53 27 51 47 C>Class(**) Lesson plan (observed) (%) 73 63 87 72 58 74 70 G>B(*), G>Class(*) Guide on how to assess 8 13 6 6 3 7 13 None students (observed) (%) Student Profiles (%) 30 20 36 33 42 34 18 U>R(*) Textbooks for classroom 56 63 48 58 41 53 67 None activities (%) Rubrics to evaluate for 6 4 11 4 11 6 6 None student work (%) Item banks for quizzes or 5 12 0 5 0 5 5 None examinations (%) Panel C – Professional Development Participated in Professional 60 65 58 59 55 59 63 None Development (%) Average Number of Days 12 12 12 12 11 12 12 None Panel D – Coaching B>Class(**), Guidance to develop their 63 63 59 69 17 63 63 G>Class(***), lesson plan (%) C>Class(***) Evaluated by the principal 86 82 89 87 85 86 87 None (%) B>Class(*), Evaluates by the principal 54 56 59 52 39 51 60 G>Class(***), on classroom practices (%) R>U(*) Evaluated by the inspector/district officer 86 84 89 85 84 85 88 None (%) Evaluates by the inspector/district officer on 59 59 63 58 45 58 61 G>Class(***) classroom practices (%) N 4,445 1,160 1,453 1,574 258 2,708 1,737 208 | The Learning Crisis in Afghanistan Table C.11: School input availability by urban/rural schools Mean diff Overall Urban Schools Rural Schools U-R Infrastructure availability 35% 38% 32% 6% Percentage of temporary classrooms 15% 16% 14% 2% Percentage of semi-permanent classrooms 15% 9% 20% -10%(**) Percentage of permanent classrooms 70% 74% 66% 8% Number of temporary classrooms 4 6 2 4(***) Number of semi-permanent 3 3 3 1 Number of permanent classrooms 15 23 8 15(***) Functioning toilets 46% 52% 41% 11% Number of Students per toilet 169 219 122 97(***) Drinking water 31% 36% 27% 10%(**) Boundary wall 80% 88% 74% 14%(**) Security guard at the door 77% 84% 71% 14%(*) Safe shelter 9% 6% 11% -5% Functioning computer 41% 60% 26% 34%(***) Internet Access 5% 6% 2% 4% Shelter (observed) 4% 4% 5% -1% Shelter (not observed) 4% 2% 6% -4% No shelter 91% 94% 89% 5% Note: The table shows the percentage of 1) Infrastructure availability, 2) Type of classrooms (permanent, semi-permanent, and temporary) 3) Functioning toilet 4)Drinking water availability 5)Boundary wall 6) Security guard at the door 7) Availability of safe shelter 8) Availability of at least one functioning computer 9)Internet access, and 10) Number of classrooms (permanent, semi-permanent, and temporary), 11) Number of students per toilet (defined as the number of students per school by the number of toilets per school) from 170 public schools in Afghanistan. The figures are disaggregated by urban and rural areas. School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 209 Table C.12: Classroom and teaching input availability by urban/rural distribution (%) Mean diff Overall Urban Schools Rural Schools U-R Functioning blackboards 89 90 87 3 Classroom visibility 79 78 81 -4 No proper seats and desks 35 32 44 -12 Chalk/marker for blackboard 99 100 97 3 Working electricity 29 33 15 19(*) Corner library 1 1 0 1 Materials displayed on walls 34 40 14 26(**) Classroom Hygiene: 8 6 13 -7 Extremely clean and well maintained Classroom Hygiene: 79 82 71 11 Reasonably clean and maintained Classroom Hygiene: 13 12 16 -5 Not very clean or maintained Share of boys in uniforms 64 67 54 13 Share of girls in uniforms 50 48 56 -8 Lesson Plan (observable) 71 74 63 11 Students with textbooks 86 87 81 6 Note: The table shows the percentage of 1) functioning blackboard, 2) classroom visibility, 3) students without proper seats and desks, 4) availability of chalk/marker, 5) working electricity, 6) availability of corner library in the classrooms, 7) whether artworks, charts, etc., displayed in classrooms, 8) classroom hygiene level, 9) share of female vs male students wearing uniform, 10) Lesson plan observed and available, 11) Share of students with textbooks, collected from a randomly selected grade 4 classroom in 170 public schools in Afghanistan, disaggregated by urban and rural areas. School level weights are used to compute these numbers. Source: SABER Service Delivery - Afghanistan 2017 210 | The Learning Crisis in Afghanistan Table C.13: Comparison of student support provided by school Boys Girls COED COED Urban Rural Statistically Schools Schools Schools Classroom Schools Schools Significant Overall (B) (G) (C) (Class) (U) (R) Difference Students with 91% 93% 85% 96% 83% 92% 89% C>G(**), C>Class(**) Pens/Pencils Students with R>U(***); B>Class(*), 59% 59% 69% 48% 85% 50% 83% Exercise Books Class>C(**) Minimum Equipment 36% 43% 30% 31% 50% 31% 50% R>U(*) Availability Pens and Exercise 38% 48% 31% 32% 50% 32% 56% R>U(**) Books Functioning 89% 92% 76% 95% 97% 90% 87% G>Class(*) Blackboard Schools with B>Class(***), Special Needs 31% 29% 43% 39% 6% 38% 25% G>Class(***), Students C>Class(***); Schools with separate classrooms for 2% 3% 4% 1% 0% 4% 0% U>R(*) Special Needs Students Schools with PTSD R>U(***); B>Class(*), 18% 22% 30% 14% 6% 33% 6% Support G>C(*), G>Class(**); Student Wrote on 55% 58% 55% 61% 40% 63% 48% None the Blackboard Teacher visited 66% 59% 74% 70% 58% 76% 57% U>R(**); student individually Teacher called Class>G(*), 46% 45% 43% 38% 65% 42% 49% student by name Class>C(**) Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 211 Table C.14: Comparison of student support available from home Boys Girls COED COED Urban Rural Statistically Schools Schools Schools Classroom Schools Schools Significant Overall (B) (G) (C) (Class) (U) (R) Difference Live with both R>U (***); Class>B(***), 36% 34% 34% 33% 58% 30% 45% parents Class>G(***), Class>C(***); R>U (***); Class>B(***), Illiterate Father 53% 48% 51% 50% 69% 46% 59% Class>G(***), Class>C(***); R>U (***); Class>B(***), Illiterate Mother 73% 71% 69% 72% 85% 68% 78% Class>G(***), Class>C(***) U>R (***); G>Class(**), Electricity 82% 81% 88% 84% 71% 88% 76% C>Class(*) U>R (***); C>Class(**); Running Water 75% 76% 81% 80% 57% 82% 69% B>Class(**), G>Class(**) Walls made of U>R (***); B>Class(***), concrete, cement 53% 55% 59% 59% 30% 70% 37% or stone G>Class(***), C>Class(***) Metal roof 32% 36% 37% 30% 24% 39% 26% U>R (***); G>Class(*) Toilet 89% 89% 93% 86% 87% 93% 85% U>R (***); G>C(**) Bed 51% 59% 57% 45% 44% 55% 48% B>C(**), B>Class(*), G>C(*); Radio 65% 65% 68% 65% 60% 67% 63% None U>R (***); G>B(**), Television 68% 66% 79% 72% 49% 78% 60% B>Class(*), G>Class(***), C>Class(***) U>R (**); B>Class(*), Refrigerator 50% 50% 59% 55% 31% 57% 44% G>Class(***), C>Class(**) Motorcycle/ Car 60% 59% 67% 58% 57% 60% 61% G>C(*) R>U (***); B>C(*), Animals 68% 69% 66% 60% 81% 59% 76% Class>B(**), Class>G(**), Class>C(***) U>R (***); G>B(*), Books 83% 82% 91% 86% 71% 89% 78% G>Class(***), C>Class(**) U>R (**); C>B(*), Mobile phone 78% 77% 80% 85% 67% 84% 74% C>Class(**); U>R (***); B>Class(***), Computer 31% 34% 35% 33% 16% 41% 22% G>Class(***), C>Class(***) Distance less than 5 5% 3% 6% 5% 6% 5% 5% G>B(*) min Distance between U>R (***); G>B(**), 56% 47% 59% 59% 57% 60% 52% 5-15 min C>B(**), Class>B(*) Distance more than R>U (***); B>G(***), 39% 50% 35% 35% 36% 35% 43% 15 min B>C(***), B>Class(**) R>U (*); B>G(**), Had Breakfast 93% 95% 88% 93% 97% 91% 95% Class>G(**) Help with 50% 55% 51% 52% 40% 53% 48% B>Class(**); homework Less than 5 90% 90% 90% 89% 90% 91% 89% None Absences Home Support 42% 48% 42% 42% 34% 44% 41% B>Class(**); Available 212 | The Learning Crisis in Afghanistan Table C.15: Comparison of dedication to learn Boys Girls COED COED Urban Rural Statistically Schools Schools Schools Classroom Schools Schools Significant Overall (B) (G) (C) (Class) (U) (R) Difference Student Absence Rate 23% 23% 15% 28% 33% 22% 28% C>G(***), Class>G(***) Student Off-task 3% 4% 3% 2% 2% 2% 4% None Time Spent Learning 1.7 1.6 1.8 1.8 1.6 1.7 1.8 None (hours) Table C.16: Student’s learning performance in language by public school/CBE schools – Grade 4 Public Schools CBE Schools Mean diff p-value Identify picture from word 64.76 82.89 -18.13 (0.02)** Score for 4 vocabulary questions 34.99 40.33 -5.33 (0.365) Vocabulary1-"farm" 33.57 39.14 -5.57 (0.346) Vocabulary2-"grow" 36.01 43.15 -7.14 (0.338) Vocabulary3-"father" 37.55 42.41 -4.86 (0.424) Vocabulary4-"on" 32.85 36.61 -3.76 (0.428) Score for 5 grammar questions 30.86 37.47 -6.61 (0.219) Grammar1-"with" 32.79 34.52 -1.74 (0.726) Grammar2-"sheep" 34.18 46.43 -12.25 (0.06)* Grammar3-"behavior" 34.26 48.36 -14.11 (0.059)* Grammar4-"third" 27.80 41.37 -13.57 (0.035)** Grammar5-"they" 25.28 16.67 8.62 (1.813) Score for 3 tenses questions 26.44 36.71 -10.26 (0.061)* Tenses1-"seek" 32.48 38.39 -5.91 (0.28) Tenses2-"went" 27.19 33.04 -5.85 (0.35) Tenses3-"is cooking" 19.66 38.69 -19.03 (0.002)*** Score for a task devoted to text 15.95 25.12 -9.17 (0.037)** comprehension Retrieve information 16.73 25.95 -9.22 (0.046)** Inference 15.17 24.29 -9.12 (0.036)** Curriculum-adjusted years of schooling 0.86 1.32 -0.47 (0.01)*** N 3,744 672 Source: SABER Service Delivery - Afghanistan 2017 Results of the Afghanistan SABER Service Delivery Survey | 213 Table C.17: Student’s learning performance in mathematics by public school/ CBE schools – Grade 4 Public Schools CBE Schools Mean diff p-value Single digit addition 77.04 70.09 6.95 (1.89) Double digit addition 49.62 54.61 -4.99 (0.59) Triple digit addition 26.95 35.27 -8.32 (0.22) Single digit subtraction 52.23 56.99 -4.76 (0.63) Double digit subtraction 22.52 37.35 -14.83 (0.01)** Triple digit subtraction 22.76 30.51 -7.75 (0.26) Single digit multiplication 40.67 41.22 -0.55 (1.02) Double digit multiplication 8.82 23.36 -14.55 (0.02)** Triple digit multiplication 6.00 12.65 -6.65 (0.19) Single digit division 30.94 51.19 -20.26 (0.01)*** Double digit division 16.60 31.55 -14.95 (0.02)** Triple digit division 14.66 17.56 -2.90 (0.62) Understand fractions 14.53 28.87 -14.34 (0.01)*** Identify a square 22.10 40.18 -18.08 (0.01)*** Solve a division word problem 15.48 38.99 -23.51 (0)*** Solve multiplication problem involving 7.51 32.74 -25.22 (0)*** monetary units Curriculum-adjusted years of schooling 1 1 0 (0.05)* N 3,744 672 Source: SABER Service Delivery - Afghanistan 2017 214 | The Learning Crisis in Afghanistan Results of the Afghanistan SABER Service Delivery Survey | 1 Systems Approach for Better Education Results saber.worldbank.org What is the real quality of education in low- and middle-income countries — not according to policy statements, but measured by what students actually experience in schools and classrooms? SABER Service Delivery (SABER SD), an initiative of the Education Global Practice, answers this question by fielding school surveys to collect data on the inputs and processes that most affect student learning. With this new data, SABER SD uncovers how well education policies are translating into results on the ground, and it flags school-level bottlenecks that need to be addressed to improve learning. The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect the views of the International Bank for Reconstruction and Development/The World Bank Group and its affiliated organizations, or those of the Executive Directors of the World Bank Group or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. 2 | The Learning Crisis in Afghanistan