79938 Kazakhstan SABER Country Report STUDENT ASSESSMENT 2012 Key Policy Areas for Student Assessment Status 1. Classroom Assessment In Kazakhstan, a publically available official document provides guidelines on classroom assessment. While the Ministry of Education and the National Academy of Education provide outlines of what students are expected to learn in different subject areas at different grade levels, the level of performance to be achieved is not clear. There are limited systematic mechanisms to monitor the quality of classroom assessment practices, and classroom assessment information is required to be disseminated only to the school director and deputy director. 2. Examinations The Unified National Testing (UNT) program is an examination administered to students in grade 11. The UNT is used as a secondary school graduation examination and a university entrance examination. Some opportunities that prepare examination staff for work on the UNT include annual non-university training courses on educational measurement and evaluation. Teachers are involved in some examination-related tasks, including the development of examination questions and scoring guides. Currently, there are no mechanisms in place to monitor the consequences of the UNT. 3. National Large-Scale Assessment (NLSA) The External Assessment of Learning Achievements (EALA) is meant to assess the quality of education services and students’ mastery of curricula at the end of basic secondary education and higher education in specific subject areas. EALA was administered for the first time in April 2012. The Government of Kazakhstan provides regular funding for core EALA activities. Some opportunities are available to prepare individuals for work on the EALA, including non-university training courses on educational measurement and evaluation, and funding for attending international programs or workshops on educational measurement and evaluation. 4. International Large-Scale Assessment (ILSA) Kazakhstan participated in TIMSS 2007 and 2011, and PISA 2009, and has plans to participate in PISA 2012, TIMSS 2015, and PIRLS 2016. A team at the National Center for Education Statistics and Assessment is responsible for carrying out ILSA activities in the country. Several ILSA team members have the necessary training and experience to carry out the required ILSA activities effectively. It is not clear whether decisions based on ILSA results have had a positive impact on students' achievement levels. THE WORLD BANK KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Introduction assessment system looks like. The framework provides guidance on how countries can build more effective Kazakhstan has focused on increasing student learning student assessment systems. The framework is outcomes by improving the quality of education in the structured around two main dimensions of assessment country. An effective student assessment system is an systems: the types/purposes of assessment activities important component to improving education quality and the quality of those activities. and learning outcomes as it provides the necessary information to meet stakeholders’ decision-making Assessment types and purposes needs. In order to gain a better understanding of the strengths and weaknesses of its existing assessment Assessment systems tend to be comprised of three system, Kazakhstan decided to benchmark this system main types of assessment activities, each of which using standardized tools developed under The World serves a different purpose and addresses different Bank’s Systems Approach for Better Education Results information needs. These three main types are: (SABER) program. SABER is an evidence-based program classroom assessment, examinations, and large-scale, to help countries systematically examine and system level assessments. strengthen the performance of different aspects of their education systems. Classroom assessment provides real-time information to support ongoing teaching and learning in individual What is SABER-Student Assessment? classrooms. Classroom assessments use a variety of SABER-Student Assessment is a component of the formats, including observation, questioning, and paper- SABER program that focuses specifically on and-pencil tests, to evaluate student learning, generally benchmarking student assessment policies and systems. on a daily basis. The goal of SABER-Student Assessment is to promote stronger assessment systems that contribute to Examinations provide a basis for selecting or certifying improved education quality and learning for all. students as they move from one level of the education system to the next (or into the workforce). All eligible National governments and international agencies are students are tested on an annual basis (or more often if increasingly recognizing the key role that assessment of the system allows for repeat testing). Examinations student learning plays in an effective education system. cover the main subject areas in the curriculum and The importance of assessment is linked to its role in: usually involve essays and multiple-choice questions. (i) providing information on levels of student learning and achievement in the system; Large-scale, system-level assessments provide feedback (ii) monitoring trends in education quality over on the overall performance of the education system at time; particular grades or age levels. These assessments (iii) supporting educators and students with real- typically cover a few subjects on a regular basis (such as time information to improve teaching and every 3 to 5 years), are often sample based, and use learning; and multiple-choice and short-answer formats. They may be (iv) holding stakeholders accountable for results. national or international in scope. SABER-Student Assessment methodology Appendix 1 summarizes the key features of these main types of assessment activities. The SABER-Student Assessment framework is built on the available evidence base for what an effective 2 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment activities; institutional The indicators are identified based on a combination of and organizational structures for designing, carrying criteria, including: out, or using results from the assessment; the • professional standards for assessment; availability of sufficient and stable sources of funding; • empirical research on the characteristics of effective and the presence of trained assessment staff. assessment systems, including analysis of the characteristics that differentiate between the System alignment refers to the extent to which the assessment systems of low- versus high-performing assessment is aligned with the rest of the education nations; and system. This includes the degree of congruence • theory—that is, general consensus among between assessment activities and system learning experts that it contributes to effective goals, standards, curriculum, and pre- and in-service assessment. teacher training. Levels of development Assessment quality refers to the psychometric quality of the instruments, processes, and procedures for the The World Bank has developed a set of assessment activity. It covers such issues as design and standardized questionnaires and rubrics for collecting implementation of assessment activities, analysis and and evaluating data on the three assessment types interpretation of student responses to those activities, and related quality drivers. and the appropriateness of how assessment results are reported and used. The questionnaires are used to collect data on the characteristics of the assessment system in a particular Crossing the quality drivers with the different country. The information from the questionnaires is assessment types/purposes provides the framework then applied to the rubrics in order to judge the and broad indicator areas shown in Table 1. This development level of the country’s assessment system framework is a starting point for identifying indicators in different areas. that can be used to review assessment systems and plan for their improvement. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, Emerging, Established, and Advanced. 3 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 These levels are artificially constructed categories substantial progress in increasing primary and secondary chosen to represent key stages on the underlying school enrollments, achieving near universal access to continuum for each indicator. Each level is basic education. However, providing quality education accompanied by a description of what performance on remains a challenge – PISA 2009 results show that 59 the indicator looks like at that level. percent of students scored below the basic competency level in mathematics, 58 percent scored below the basic • Latent is the lowest level of performance; it competency level in reading, and 55 percent scored below represents absence of, or deviation from, the the basic competency level in sciences. desired attribute. • Emerging is the next level; it represents partial As of 2011, Kazakhstan has reached near universal presence of the attribute. primary and secondary enrollment rates, although • Established represents the acceptable minimum tertiary enrollment rates are lower than those of high- standard. performing education systems. Progression indicators • Advanced represents the ideal or current best are strong: the primary completion rate, lower practice. secondary graduation ratio, and tertiary graduation ratio are high relative to comparator countries. A summary of the development levels for each assessment type is presented in Appendix 3. The Government of Kazakhstan is committed to investing in human capital as illustrated in the country’s National In reality, assessment systems are likely to be at Strategic Plan 2020 in which human resources development different levels of development in different areas. For is one of the five pillars. Moreover, the National Strategy for example, a system may be Established in the area of Industrial Innovation Development includes as one of its examinations, but Emerging in the area of large-scale, objectives a tripling of labor productivity between 2000 and system-level assessment, and vice versa. While intuition 2015. The Government views modernization of the suggests that it is probably better to be further along in education system as a way to achieve these goals. as many areas as possible, the evidence is unclear as to whether it is necessary to be functioning at Detailed information was collected on Kazakhstan’s Advanced levels in all areas. Therefore, one might view student assessment system using the SABER-Student the Established level as a desirable minimum outcome to Assessment questionnaires and rubrics. It is important achieve in all areas, but only aspire beyond that in those to remember that these tools primarily focus on areas that most contribute to the national vision or benchmarking a country’s policies and arrangements for priorities for education. In line with these considerations, assessment activities at the system or macro level. the ratings generated by the rubrics are not meant to be Additional data would need to be collected to additive across assessment types (that is, they are not determine actual, on-the-ground practices in meant to be added to create an overall rating for an Kazakhstan, particularly by teachers and students in assessment system; they are only meant to produce an schools. The following sections discuss the findings by overall rating for each assessment type). The methodology each assessment type, accompanied by suggested for assigning development levels is summarized in policy options. These sections present the analysis of Appendix 4. information collected using the SABER-Student Assessment questionnaires and rubrics that is also Education in Kazakhstan described in “Student Assessment System Policy Note: Kazakhstan 2012.� The suggested policy options were Kazakhstan is an upper middle income country that is determined in collaboration with key local stakeholders experiencing high economic growth, with GDP per capita based on Kazakhstan’s immediate interests and needs. of $11357 (2011), and annual growth of approximately 6 Detailed, completed rubrics for each assessment type in percent. To meet the needs of its expanding economy, Kazakhstan are provided in Appendix 5. Kazakhstan recognizes the need to increase skill levels and learning outcomes through improving the quality of education, among other strategies. Kazakhstan has made 4 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Classroom Assessment Classroom assessment practices in Kazakhstan are of moderate quality. Classroom assessment practices are generally aligned with the pedagogical and curricular Level of development frameworks, and usually provide useful feedback to students. Parents are usually well informed about In Kazakhstan, teachers primarily carry out classroom students' grades. However, classroom assessment assessment activities to meet external requirements or activities tend to be mainly about recalling information. information needs, to inform their own teaching and Data about student performance on classroom their students’ learning, and to meet school-level assessment activities are mainly used as an requirements or information needs. Clear guidelines for administrative or control tool, rather than as a classroom assessment are outlined by a formal, pedagogical or curricular tool. Scoring is usually fair: publically available system-level document, Model rules errors in the scoring of students' work, uneven for monitoring current performance progress, interim, application of standards for grading students’ work, and and final attestation of students at institutions grade inflation are rarely serious problems. Limited delivering the general education program at the systematic mechanisms exist to monitor the quality of primary, basic secondary, and general secondary classroom assessment practices. For example, education levels, a 2008 decree that was revised in classroom assessment is a required component of a December 2009. teacher’s performance evaluation and of school inspection and teacher supervision. Some system-wide resources are available to teachers to carry out classroom assessment activities, including Kazakhstan also makes efforts to ensure effective uses resources that outline what students are expected to of classroom assessment. Teachers are required to learn in different subject areas at different grades or report individual student performance to the school age levels. Specifically, the guidelines in the State director and deputy director. Students’ classroom compulsory standard for primary, basic secondary, and assessment results are recorded in the students’ own general secondary education document and the copy books and teacher record books. There are also Instructional and methodological guidelines on specifics adequate required uses of classroom assessment to of teaching in various subject areas at a given level of support student learning. Classroom assessment education explain education process requirements at information is not used as an input for external each level of primary and secondary education, outline examination results. Required uses of classroom learning objectives for a particular subject at a given assessment activities to promote and inform student grade level, specify what students are expected to learning include diagnosing student learning issues, learn, and offer brief recommendations on the various providing feedback to students on their learning, forms of classroom assessment activities (theme�based informing parents about their child’s learning, planning written exams, oral quizzes, and testing, among others). next steps in instruction, and grading students. However, the recommendations are only limited to enumerating the format of assessment activities Suggested policy options: without a detailed explanation for use of assessment information or the application of scoring criteria. 1. Better alignment of classroom assessment with system learning goals. This can be done by In Kazakhstan, there are few system level mechanisms introducing and making available to teachers a to ensure that teachers develop skills and expertise in variety of system-wide resources for carrying classroom assessment. There are no required courses out classroom assessment activities, including on classroom assessment included in the pre- or in- an official curriculum or standards document service training programs offered by the National that outlines the level(s) of performance that Institute of Continuing Education for Managers, students are expected to reach in different Academic, and Teaching Staff, or by the regional subject areas at different grade/age levels, Teacher Training Institutes of Continuing Education. textbooks or workbooks that provide support 5 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 for classroom assessment, scoring criteria or rubrics for students' work, and online Examinations assessment resources. 2. Ensure that teachers develop skills and expertise Level of development in classroom assessment. This can be done by introducing varied system-level mechanisms Kazakhstan’s national examination, the Unified National such as pre- and in-service teacher training on Testing (UNT) program, is administered to students in classroom assessment, providing a required grade 11 to serve as both student certification for course on classroom assessment in all teacher school cycle completion and to select students to training programs, and making available university or other higher-education institutions. The opportunities for teachers to participate in Committee for Quality Control in Education and Science conferences and workshops. and the National Testing Center (NTC) under the Ministry of Education and Science of the Republic of 3. Ensure the quality of classroom assessment by Kazakhstan are responsible for implementing the UNT. putting in place varied and systematic The examination includes four compulsory subjects mechanisms to monitor the quality of classroom (language of instruction, Kazakh language for schools of assessment practices. These practices could Russian instruction and Russian language for schools of include introducing an external moderation Kazakh instruction, mathematics, and history of system that reviews the difficulty of classroom Kazakhstan) and one optional subject (physics, assessment activities and the appropriateness chemistry, biology, geography, world history, Kazakh of scoring criteria. Additionally, system-wide literature, Russian literature, English language, German reviews of the quality of education that include language, French language, or arts). a focus on classroom assessment can be conducted, and funding for research on the Efforts to reform the UNT are ongoing. In 2011, the quality of classroom assessment activities and Government of Kazakhstan improved UNT test how to improve classroom assessment can be specifications for certain subjects in which students provided as well. previously scored poorly, and in 2012, efforts were made to improve the quality of some UNT items, such 4. Ensure the effective uses of classroom as those focused on assessing logical thinking in assessment information. This could be done by mathematics. The cutoff score was also raised, which requiring classroom assessment information to may increase the number of students failing to reach be disseminated to all key stakeholders, this cutoff on the exam. Currently, the UNT is paper- including students, parents, and education based, but in 2013, the Government of Kazakhstan officials, as well as by using classroom plans to introduce computer-based testing. In 2013, the assessment information as an input to external Government will also pilot test two separate exams for examinations. the UNT: a school-leaving examination and a university entrance examination. Clear policies for the UNT examination are in place and the UNT has been administered since 2004. However, the number and content of subjects tested has changed over time, which makes it difficult to track trends in results. The UNT is authorized by two formal documents: State Program for Education Development 2011-2020 and The Rules on Administering the Unified National Testing. These documents are available to the public and address some key aspects of the 6 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 examination, including governance, distribution of errors in scoring, weaknesses in test design, and power, responsibilities among key entities, and funding frequent errors in the examination questions. sources. The UNT is aligned with learning goals and students’ There are independent attempts to improve the opportunities to learn to a certain degree. There is a examination by stakeholder groups, including the clear understanding of what the UNT measures. Ministry of Education, National Center for Education Information on the format of the UNT examinations and Statistics and Assessment, National Center for State sample tests are published on the Committee for Educational Standards and Testing, and the Committee Quality Control in Education and Science’s website. for Quality Control in Education and Science. New Study guides that include sample test items and organizational arrangements have been introduced to preparation advice are also available for purchase address security, cheating, and efficiency issues. In throughout the country. Some students may have addition, test development staff from the National greater access than others to these preparation Center for Education Statistics and Assessment received materials depending on access to the internet and the technical assistance under the World Bank Joint availability of funds to purchase paperback study Economic Research Program (JERP) to improve the guides. quality of UNT items in accordance with international standards. Experts at the Center are also working on Some stakeholder groups, including students and new ability tests to measure students’ higher-order parents, question what the examination measures. The competencies and skills. most sensitive issue often raised by parents and students, as well as by the media, is the inadequate Regular funding is allocated to cover all core quality and validity of the test items. Too many items examination activities of the UNT: design, seem to rely on rote memorization. In a recent survey administration, data analysis, data reporting, and staff conducted by “Kazakhstan Today,� the vast majority of training. This regular funding does not currently cover respondents believed that the UNT is not an objective research and development activities. assessment of students’ knowledge. 1 Additional reporting by “Kazpravda� also mentions this criticism The Committee for Quality and Control in Education and against the exam. 2 The Government of Kazakhstan Science and the NTC has all of the required facilities to recognizes the need to improve the quality of the UNT, carry out the examination, including computers for all and is addressing some of these concerns with technical staff, secure storage facilities, and adequate forthcoming “ability tests.� In 2013 and 2014 the communication tools. However, the Committee and the Ministry of Education plans to pilot a two stage UNT NTC are newly established as of 2004, and not examination: the National (school-leaving) Exam, which accountable to an external board or agency. In addition, is intended to test students’ functional literacy and UNT results are not recognized by other countries, only logic, and the Complex Exam (university entrance), by certification and selection systems in Kazakhstan. which is to be an examination for entrance to higher education institutions with various subject tests that are The examination office is inadequately staffed to aligned with possible courses of study. effectively carry out the UNT. The country offers limited training opportunities related to examination development, including annual non-university training courses on educational measurement and evaluation, and funds attendance at similar international programs. Although the Ministry of Education and subnational governments ensure that each UNT testing site is 1 Kazakhstan Today (2012). Available online: adequately staffed, effective administration has been a http://www.kt.kz/?lang=rus&uin=1133168071&chapter=115 challenge due to poor training of the test 3551658 administrators, unclear instructions and guidelines, 2 Kazpravda (2012). Available online: http://www.kazpravda.kz/c/1316743767 7 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Teachers are provided with opportunities to learn about transparent entrance to higher education institutions. A the UNT. There are up-to-date compulsory courses or more equitable method for selection to higher workshops on examinations for teachers, and teachers education institutions was in fact one of the main goals are involved in some examination-related tasks, such as when the Government of Kazakhstan set up the UNT selecting or creating examination questions and scoring system. Moreover, there are no mechanisms in place to guides, and supervising examination procedures. monitor the consequences of the examination. Analysis of results is usually descriptive in nature, and a more There are some procedures and policies in place to thorough review could yield findings on, among other ensure the quality of the UNT. There is a comprehensive things, the impact on a candidate of failing the technical report with restricted circulation – the NTC examination, and the predictive validity of the exam. prepares test specifications, which are approved by the Ministry of Education, but these are not made available Suggested policy options: to the general public. In addition, internal reviews and observers are used to ensure the quality of the 1. Ensure that the examination program is supported examination. by effective human resources. It is very important that there is support for the training and All students can take the examination regardless of professional development of staff in the background, location, or ability to pay. However, the examination office so that they can carry out results are not credible to some stakeholder groups, examination activities more effectively, and with including parents and students. no issues. This can be done by introducing a range of opportunities that prepare individuals for work Inappropriate behavior surrounding the examination on the UNT, including university graduate process is high. Documented cases have involved programs (masters or doctorate level) and leakage of examination content prior to the university courses (graduate and non-graduate) examination, impersonation of another individual, on educational measurement and evaluation, as unauthorized communication with other candidates by well as internships in the examination office. mobile phone or other method, use of prohibited materials, intimidation of examination supervisors, and 2. Ensure the quality of the UNT. One way to better other problems. ensure quality is to make the UNT technical report available to the general public. Other options Student names and results are not confidential, as include introducing more systematic quality students’ UNT scores are published at each UNT control and assurance mechanisms, such as administration site, in two national newspapers, and on external observers, external certification or audit a government website. of the UNT program, and piloting or field testing UNT questions. There are very limited options for students who do not perform well on the examination. Based on the UNT 3. Ensure fairness of the UNT by reviewing the results, students with the highest scores receive procedures for preventing and addressing government scholarships to enter top higher education inappropriate behavior, and putting in place institutions that require the highest UNT scores; others mechanisms to ensure that inappropriate behavior who pass the cutoff score may choose less competitive does not take place. Ensuring that the UNT is a fair schools or fee-based tuition. The rest of the students and just exam system is a priority of the who do not pass the cutoff score are left to pass the Government of Kazakhstan. Thus, establishing UNT again in another year or enter education public monitoring committees and introducing institutions abroad, including Russia, China, and Eastern electronic testing instruments and computer Europe. According to a recent opinion poll taken in programs to conduct analyses of UNT results will Kazakhstan, 53 percent of people believe that the assist in preventing the occurrence of inappropriate existing UNT system is not effective in allowing fair and behavior, academic misconduct, and dishonesty. 8 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 National Large-Scale Assessment (NLSA) annually in Kazakh language and three additional subjects to all students at the final grades of lower secondary, upper secondary, and university education Level of development Most aspects of setting clear policies for the EALA are From 2005 to 2011, the Interim State Control (ISC) strong, even though the EALA has only been assessment was administered annually to all students in administered once. There is a formal and publicly grades 4 and 9 across a rotating set of subjects, available policy—the Law of the Republic of Kazakhstan including mathematics, native language, foreign On Education—that authorizes the EALA and establishes language, geography, World Knowledge, physics, it as a form of independent monitoring of the quality of biology, history of Kazakhstan, and chemistry. The ISC education. There is also a written plan for implementing had four objectives: (i) to carry out an assessment of the EALA, which provides detailed organizational and students’ learning achievements; (ii) to evaluate the implementation arrangements. effectiveness of the learning process; (iii) to develop recommendations for improvement of state compulsory There is regular funding allocated to the EALA to cover education standards; and (iv) to carry out a comparative all core assessment activities, including design, analysis of the quality of educational services provided administration, analysis, and reporting. This funding by institutions. anticipates medium- and long-term planning of EALA programs and staff training, but it does not cover In 2012, the External Assessment of Learning research and development activities. Achievements (EALA) was introduced to replace the ISC 3 in response to the adoption of the State Program for Although the EALA is a new assessment, strong Education Development 2011-2020. The EALA aims to organizational structures are in place. There are assess the quality of education services and the level of permanent agencies associated with the NLSA – the students’ mastery of curricula at the end of basic Ministry of Education’s National Testing Center (NTC) secondary education and higher education. The main develops and administers the EALA, and the Division of purposes of the EALA are to: (i) monitor education External Examinations develops related policies. In quality at the system level; (ii) hold schools and addition, the NTC is accountable to a clearly recognized educators accountable; and (iii) inform policy design, body, the Ministry of Education’s Committee for Quality evaluation, and decision-making. The first EALA in 2011 Control in Education and Science. assessed 10 percent of students in Grade 9 in four subjects: language of instruction (Kazakh or Russian), Kazakhstan also has effective human resources for the history of Kazakhstan, algebra, and chemistry. The EALA. The country offers some opportunities to prepare majority of students took a paper-and-pencil test, but individuals for work on the NLSA, including non- students in the capital city, Astana, and in the Atyrau university training courses on educational measurement region took a computer-based test. and evaluation, and funding for individuals to attend similar international programs. For example, the The EALA will be adjusted as the Government of National Center for Education Statistics and Assessment Kazakhstan begins a transition to a 12-year school offers professional training courses in the area of model. The Government plans to administer the EALA education quality monitoring, management, and measurement. In Kazakhstan, the EALA is aligned with learning goals, although the degree of acceptance of its results by stakeholder groups remains to be seen. The EALA measures performance against learning standards 3 The EALA is similar to the ISC, but educational institutions related to the mastery of basic secondary and general are not subject to closure based on the results of the EALA. secondary education programs. There are also regular The ISC was abolished in 2011 by Presidential order. 9 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 internal reviews of the EALA to ensure that it measures Suggested policy options: what it intends to measure. Test items are developed by a team of experts (including qualified secondary 1. Ensure the availability of effective human teachers, candidates and doctors of the sciences, resources for carrying out EALA activities. This university professors, and others), and then reviewed could be done by, for example, introducing by secondary school teachers. varied systematic mechanisms to prepare individuals to work on EALA. Such mechanisms Teachers are provided with some opportunities to learn might include introducing university graduate about the EALA through courses or workshops that are programs and courses specifically focused on offered on a regular basis. In addition, the NTC regularly educational measurement and evaluation, and holds scientific and practical workshops in the area of making available internships or short-term test development to provide methodological assistance employment at the NTC and the National to item writers and reviewers (mostly secondary school Center for Education Statistics and Assessment. teachers). 2. Ensure the quality of the EALA by introducing a The EALA has specific procedures in place to ensure variety of mechanisms specifically for this quality, including training proctors and administrators, purpose. Mechanisms might include appointing providing administrators with a standard manual, internal reviewers or observers, introducing piloting the administration of the assessment, tracking external certification or audit, and training booklets with serial numbers, and using external scorers to guarantee high inter-rater reliability. observers. There are plans to establish a working group in 2012 to improve the EALA procedures. In addition, 3. Ensure that EALA results are communicated and the EALA has a comprehensive technical report with used by all stakeholders. Often, reports on the restricted circulation, which meets the basic findings of NLSA exercises, such as the EALA, requirements of NLSA documentation. can be very technical and not relevant for a broad range of stakeholder groups. Thus, a Related to equity, different assessment options are variety of reports and communication offered to include all groups of students in the EALA, mechanisms should be used to better address including students in hard-to-reach areas and students the diverse information needs of stakeholders. who have a different language of instruction – the test Examples of reports and communication is administered in Kazakh or Russian, depending on the mechanisms include customized summary student’s language of instruction. The next EALA will be reports for policy makers; targeted reports for held at 154 sites across the country to ensure that schools, teachers, curriculum developers, and students do not have to travel far to participate. teacher trainers; as well as press releases and special reports for radio and television. The EALA was administered for the first time on April 20 Adequate budget will need to be allocated for and 23, 2012, therefore limited information is available EALA report preparation and dissemination. regarding uses of the NLSA results. However, in the future, it is important that the EALA continues to be disseminated in an effective way, and used by at least some stakeholder groups in a way that is consistent with the purposes and technical characteristics of the assessment. There should also be some mechanisms in place to monitor the consequences of the EALA. 10 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 International Large-Scale Assessment The ILSA team has been unable to attend all of the international workshops or meetings for the ILSA (ILSA) exercises in which the country has participated. This is mainly due to a lack of funding. Level of development To date, Kazakhstan has met all of the technical standards required to have its data presented in the Kazakhstan has set clear policies for ILSA activities. main displays of the international report for each ILSA Students in Kazakhstan have participated in three ILSAs exercise in which it has participated. However, in the last 10 years: the Trends in International Kazakhstan has not contributed new knowledge on ILSAs Mathematics and Science Study (TIMSS) 2007 (grade 4); to the global evidence base. TIMSS 2011 (grades 4 and 8); and the Program for International Student Assessment (PISA) 2009. The Kazakhstan’s ILSA results and information are regularly Government of Kazakhstan has also taken concrete steps disseminated throughout the country in a number of to participate in three upcoming ISLAs: PISA 2012, TIMSS ways. Results for TIMSS 2007 were included in the 2009 2015, and PIRLS 2016. Participation in these ILSAs is and 2010 versions of the National Report on the Status addressed by a formal public policy document, State of Education in Kazakhstan, and the results were also Program for Education Development of the Republic of published in a TIMSS national report and communicated Kazakhstan 2011-2020. through media outlets. In addition, products to provide feedback to schools and educators about the ILSA results There is regular funding for ILSA exercises, allocated at were sometimes made available. The National Center for the discretion of the government. The funding typically Education Statistics and Assessment usually works with covers all core activities of the ILSA, including subnational departments of education to disseminate international participation fees, implementation of the results, who in turn work with schools and educators. assessment exercise in the country, processing and There is some media coverage of the results in major analyzing assessment data, reporting and disseminating newspapers, information agencies, and on government the results in Kazakhstan, and attending relevant websites. Although PISA 2009 results have motivated a international expert meetings. Funding does not cover five-year national work plan on functional literacy research and development activities. development, ILSA results are used in a limited way to inform decision making in Kazakhstan. It also is not clear The human resources needed for ILSA are partially in that decisions based on ILSA results have had a positive place. There is a team and national coordinator to carry impact on students’ achievement levels. out ILSA activities under the Director of the National Center for Education Statistics and Assessment. In Suggested policy options: addition, regional coordinators oversee the organization and implementation of the assessment in each region. 1. Ensure sustainability of the ILSA program. This The national coordinator is fluent in the language of the can be done by including implementation of assessment. However, the ILSA office is inadequately international assessments as a line item in the staffed to carry out these assessments in a completely regular government budget, and by ensuring effective manner. Some of the team members have the that the funding provided is sufficient to cover necessary training, but most of them do not have research and development as well as core ILSA extensive training or experience in the area of activities. international assessments. 2. Provide learning opportunities on ILSA as a Kazakhstan provides some opportunities to learn about professional/technical field. This can be done by ILSAs, but only to the ILSA team members; there are providing opportunities to learn about ILSA for training workshops, but no university or online courses university students studying assessment, or a on the topic of international assessments. related area, and for professionals or university 11 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 staff interested in assessment. Learning opportunities could include university courses on the topic of international assessments, funding for attending international workshops or training on international assessments, and on-line courses on international assessments. 3. Ensure effective uses of ILSA results. This can be done by regularly and widely disseminating the country-specific ILSA results and information in Kazakhstan as well as by making customized products that provide feedback about ILSA results to schools and educators. 12 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations Surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education system opportunities instruction grade/age system at to the next (or level(s), and to particular into the monitor trends in grade/age workforce) learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the system where the system basis (such as basis (such as allows for allows for every 3-5 years) every 3-5 years) repeats repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a few confined to one subject areas subject areas subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques 13 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses 14 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide institutional wide institutional support and ensure the capacity to support and capacity to support and capacity to support and quality of classroom ensure the quality of ensure the quality of ensure the quality of assessment practices. classroom assessment classroom assessment classroom assessment practices. practices. practices. CLASSROOM ASSESSMENT There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and and a need to develop There is institutional institutional capacity and institutional capacity to capacity and some strong mechanisms to run the examination. The limited mechanisms to monitor it. The EXAMINATIONS examination typically is monitor it. The examination is of high of poor quality and is examination is of quality and is perceived perceived as unfair or acceptable quality and is as fair and free from corrupt. perceived as fair for corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA place. NLSA in place and a in place. There is in place and institutional need to develop institutional capacity and capacity and strong institutional capacity to some limited mechanisms to monitor run the NLSA. mechanisms to monitor it. The NLSA is of high NATIONAL (OR SYSTEM- Assessment quality and it. The NLSA is of quality and its LEVEL) LARGE-SCALE impact are weak. moderate quality and its information is ASSESSMENT information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is need to ILSA. There is and institutional capacity in one. develop institutional institutional capacity to to run the ILSA. The capacity to carry out the carry out the ILSA. The information from the INTERNATIONAL LARGE- ILSA. information from the ILSA is effectively used SCALE ASSESSMENT ILSA is disseminated, to improve education. but not always used in effective ways. 15 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 4: Methodology for Assigning Development Levels 1. The country team or consultant collects information 5. The preliminary development level is validated using about the assessment system in the country. expert judgment in cooperation with the country team and The World Bank Task Team Leader. 2. Based on the collected information, a level of development and score is assigned to each dimension in For scores that allow a margin of discretion (i.e., to the rubrics: choose between two levels of development), a final decision has to be made based on expert judgment. For • Latent = 1 score point example, the aforementioned hypothetical country has • Emerging = 2 score points an ‘Enabling Context’ score of 2.33, corresponding to a • Established = 3 score points preliminary level of development of ‘Emerging or • Advanced = 4 score points Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most 3. The score for each quality driver is computed by appropriate level. aggregating the scores for each of its constituent dimensions. For example: 6. Scores for certain key dimensions under ‘Enabling Context’ (in the case of EXAM, NLSA, and ILSA) and The quality driver, ‘Enabling Context,’ in the case of under ‘System Alignment’ (in the case of CLASS) were ILSA, has 3 dimensions on which a hypothetical country set as ceiling scores, i.e., the overall mean score for the receives the following scores: Dimension A = 2 points; particular assessment type cannot be greater than the Dimension B = 2 points; Dimension C = 3 points. The score for these key dimensions. These key variables hypothetical country’s overall score for this quality include formal policy, regular funding, having a driver would be: (2+2+3)/3 = 2.33 permanent assessment unit, and the quality of assessment practices. 4. A preliminary level of development is assigned to each quality driver. 16 Systems Approach for Better Education Results KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 5: SABER-Student Assessment Rubrics for Kazakhstan This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in Kazakhstan. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection (as indicated by a thick border and an asterisk). The explanation or justification text can be located in the “Development level rating justifications� section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. 17 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 KAZAKHSTAN Classroom Assessment 18 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. 1 assessment. classroom assessment. assessment. * 2 This option does not apply to this This option does not apply to this The availability of the document is The document is widely available. dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide 3 teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to to what level of performance. level of performance required is not what level of performance. * 4 clear. ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom * 5 assessment. assessment. 19 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are 6 from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. There are no mechanisms to monitor the There are ad hoc mechanisms to monitor * There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the 7 practices. practices. assessment practices. quality of classroom assessment * practices. ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 8 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support student learning. student learning, excluding its use as an student learning, including its use as an 9 input for external examination results. * input for external examination results. 20 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Classroom Assessment: Development-level rating justifications 1. The Model rules for monitoring current performance progress, interim and final attestation of students at institutions delivering general education programs at the primary, basic secondary and general secondary education levels is a formal document authorized by the Ministry of Education and Science of the Republic of Kazakhstan by the Decree of the Minister of Education # 125 dated March 18, 2008, and # 590 with additions and amendments as of December 25, 2009. 2. The document is available online, in public libraries, in in-service teacher training courses, at education institutions, and sub-national departments of education. 3. A document that outlines what students are expected to learn in different subject areas at different grade/age levels is typically available to teachers for their classroom assessment activities. The State compulsory standard for primary, basic secondary, and general secondary education document provides guidelines to education process requirements at each level of education (primary, basic secondary, and general secondary). Additionally, the Ministry of Education and National Academy of Education jointly develop "education programs" which provide a more detailed outline of what students are expected to learn in different subject areas at different grade levels. Specialized education programs for talented students and students with special needs are also developed. Education programs, however, do not outline the level(s) of performance that students are expected to reach in different subject areas at different grade/age levels. The instructional and methodological guidelines on specifics of teaching in various subject areas at a given level of education outline learning objectives for a particular subject at a given grade level, provide the list of recommended textbooks, specify what students are expected to learn and occasionally provide brief recommendations on the various forms of assessment that may be used in the classroom, such as theme-based written exams, oral quizzes, testing etc. However, recommendations are only limited to enumerating the format of assessment activities without any detailed explanation for their uses, expectations, scoring criteria etc. 4. The document outlines what student at different grade/age levels are expected to learn, but does not specify to what performance level. 5. Opportunities to participate in item development for, or scoring of, large-scale assessments or exams are available at the system level to ensure that teachers develop skills and expertise in classroom assessment. The National Institute of Continuing Education for Managers, Academic and Teaching Staff carries out and ensure the quality of continuing education and retraining of staff at the secondary, general and primary professional education, improves professional competences of managers’ and teachers’ corps, and monitors the needs in further training and retraining of teachers in Kazakhstan. 16 regional Teacher Training Institutes of Continuing Education provide regular pre- and in-service training programs for teachers, develop textbooks, methodological, information and other documents and promote their practical implementation, study teachers’ training needs, carry out expert evaluation of curricula, learning materials etc. However, there is no training that specifically targets classroom assessment and there are no required courses on classroom assessment. 21 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 6. Classroom assessment practices are generally aligned with pedagogical and curricular framework. It is common to observe classroom assessment practices to rely mainly on multiple-choice, selection-type questions, to be mainly about recalling information, and to mainly be used as administrative or control tool rather than as pedagogical resource. It is rare to observe errors in the scoring or grading of students' work. Additionally, uneven application of standards for grading students’ work is rarely a serious problem. Grade inflation is rarely a serious problem and parents are usually well informed poorly informed about students' grades. Classroom assessment practices usually provide useful feedback to students. 7. Classroom assessment is a required component of a teacher’s performance evaluation and of school inspection or teacher supervision. 8. Teachers are required to report to the school director and deputy director on individual student’s performance. 9. Required uses of classroom assessment activities to promote and inform student learning include diagnosing student learning issues, providing feedback to students on their learning, informing parents about their child’s learning, planning next steps in instruction, and for grading students for internal classroom uses. 22 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 KAZAKHSTAN Examinations 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this 1 place. operating on an irregular basis has been operating regularly. dimension * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this 2 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. This option does not apply to this The policy document is not available to * The policy document is available to the This option does not apply to this 3 dimension. the public public. dimension. This option does not apply to this This option does not apply to this The policy document addresses some * The policy document addresses all key 4 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 5 the examination or are indifferent to it. examination. examination. examination. There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder 6 groups. groups. * Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. 7 charge of the examination. charge of the examination. (CONTINUED) 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 8 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. 9 processing or reporting. processing and reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and 10 dimension. development. dimension. development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this 11 is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this accountable to an external board or dimension. an external board or agency. dimension. 12 agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection 13 country. another country. system in another country. * The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. 14 examination. examination. examination. * (CONTINUED) 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 15 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on 16 examination. examination. the examination. * 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 17 measures. dimension. the examination measures. dimension. What the examination measures is This option does not apply to this * What is measured by the examination is This option does not apply to this questioned by some stakeholder dimension. largely accepted by stakeholder groups. dimension. 18 groups. * Material to prepare for the examination There is some material to prepare for the There is comprehensive material to There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 19 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for 20 teachers. to teachers. teachers. Teachers are excluded from all Teachers are involved in very few Teachers are involved in some Teachers are involved in * most 21 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality 22 documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the 23 examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the 24 examination process is high. examination process is moderate. examination process is low. examination process is marginal. * The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this 25 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other 26 language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. barriers. equivalent barriers. equivalent barriers. * (CONTINUED) 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all 27 proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. 28 Student names and results are public. This option does not apply to this Students’ results are confidential. This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform well on the do not perform well on the examination. who do not perform well on the 29 or students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the 30 examination. examination. examination. * 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Examinations: Development-level rating justifications 1. The Unified National Testing (UNT) is an examination administered to students in grade 11. The examination is used as a secondary school graduation exam and a university entrance exam. The examination includes compulsory and optional subjects. From 2004 – 2007, 3 compulsory subjects (Language of instruction (Kazakh or Russian), Mathematics, History of Kazakhstan) and 1 optional subject were tested. From 2008 – 2011, 4 compulsory subjects (Native language (language of instruction), Mathematic s, History of Kazakhstan, Kazakh language for schools with Russian language of instruction and Russian language for schools with Kazakh language of instruction) and one optional subject were tested. Optional subjects include: Physics, Chemistry, Biology, Geography, World History, Kazakh Literature, Russian Literature, English language, German language, French language, and arts disciplines. In 2013, in addition to introducing computer-based testing, the Government of RK plans to pilot test the separation of UNT functions into two exams (a school-leaving examination and a university entrance examination), scale up this effort by 2014, and finalize the separation in 2015 with extensive use of computer-based testing. Applicants to higher education institutions will have to sit for additional subject exams in the form of independent national testing. 2. The "State Program for Education Development 2011-2020� formal policy document, authorized by the Decree of the President of the Republic of Kazakhstan № 1118 dated December 7, 2010 (Government of the Republic of Kazakhstan), authorizes the examination. The Minister of Education also approved “The Rules on administering the Unified National Testing� (The Rules) policy document. This document is also available to the public. 3. The policy document is publicly available for download at the Ministry of Education and other government agencies’ websites. 4. The "State Program for Education Development 2011-2020" document outlines governance, distribution of power, responsibilities among key entities, and states funding sources. "The Rules" document specifies, among other procedures, who can sit for the examination, alignment of the examination with curricula and standards, outlines procedures and consequences of cheating and other forms of inappropriate behavior, work of the Review Board, as well as general procedures for disadvantages/special/students with disabilities. 5. Limited information is available support the selection for this dimension. 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 6. Independent efforts by different stakeholder groups have been made to improve the examination. The Ministry of Education and it's affiliated agencies (National Center for Education Statistics and Assessment, National Center for State Educational Standards and Testing (National Testing Center), Committee for Quality Control in Education and Science) has made coordinated efforts to improve the examination. In addition, in 2012, new organizational arrangements primarily for addressing issues of security, cheating, and efficiency, will be introduced during administration of the UNT. These arrangements include cell phone signal suppressors, videotaping equipment, decrease in the number of exam rooms at each UNT administration site, decrease in the number of State Committee members who are responsible for organizational arrangements during the exam, informational and outreach campaigns among population. Technical assistance is provided by World Bank experts as part of the Kazakhstan-World Bank Joint Economic Research Program to improve the quality of test items based on international standards. World Bank experts worked with test development staff at the National Center for Education Statistics and Assessment under the Ministry of Education (The Center) which houses two laboratories on test development and expert evaluation to deliver capacity building training workshops on item writing and analysis, peer reviewing, score interpretation and other aspects of test design. In addition and continuation of this capacity building exercise, the experts at the Center are currently working further on developing the so-called “tests of new generation� (ability tests) to measure students’ higher-order competencies and skills. 7. Limited information is available support the selection for this dimension. 8. There is regular funding allocated by the government. 9. Funding covers all core examination activities. 10. Funding does not cover research and development. 11. The Committee and the NTC have been operating since 2004. 12. The Committee and the NTC are accountable to the Ministry of Education and Science of RK, Government of RK, and President’s Administration only. They do not report to external independent bodies. 13. Examination results are officially recognized by certification and selection systems in the country. 14. The Committee and the NTC have computers for all technical staff, secure storage facilities, and adequate communication tools (phone, email, internet, etc.). The buildings are also generally secure and there is generally adequate access to computer servers. 15. The Ministry of Education and sub-national governments ensure that each UNT site is adequately staffed. The general procedure for the number and type of staff required is reflected in the Rules on administering the UNT. Issues that have been identified with the performance of the human resources that are responsible for the UNT include poor training of test 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 administrators or about unclear instructions and guidelines in administering the examination, errors in scoring that have led to delays in results being reported, weaknesses in test design, and frequent errors in the examination questions. Generally, the most sensitive issue often raised by parents and children as well as media is the perceived inadequate quality and validity of the test items. 16. Non-university training courses/workshops on educational measurement and evaluation are available in the country on an annual basis. Additionally, funding for attending international programs, courses, and workshops on educational measurement and evaluation are available in the country as well. 17. The examination measures national school curriculum guidelines and standards. 18. Some stakeholder groups question what the examination measures. Students and parents are generally concerned with the perceived inadequacies of the test items as many items test what is viewed as irrelevant factual knowledge in a particular discipline. The Government of RK recognizes these concerns and revises the item bank on a regular basis. Test development experts at the Center under the Ministry of Education of RK are working on the new tests that would measure students’ higher-order competences and skills (so-called “ability tests�). 19. The needed material to prepare for the examination is accessible by most students (50% to 90% of students), but certain student subgroups may have greater access than others (e.g., due to language issues, location). Specifically, while sample tests are available for download at the Ministry’s website, not all students have free access to internet or financial resources to purchase paperback study guides. The material on the examination that is publically available includes examples of the types of questions that are on the examination and a report on the strengths and weaknesses in student performance. Information on the format of the examination and sample tests are published on the Ministry of Education website, Committee for Quality Control in Education and Science. UNT study guides are also available for purchase throughout the country and usually include sample test items in relevant disciplines from the actual tests and preparation advice. Annually, the Center prepares and publishes the National Report on the Status of Education in RK. One section of the report presents brief aggregated results of the UNT, including strengths and weaknesses in student performance. However, the presentation of the results is generally limited to basic descriptive statistics. 20. There are compulsory courses or workshops on the examinations available to teachers that are regularly updated. National Testing Center, based on recommendations of departments of education, holds training workshops for school teachers. These workshops include training by international test development experts. 21. Examination-related tasks that are mainly performed by teachers include selecting or creating examination questions and examination scoring guides, and until 2012, supervising the examination procedures. Teachers in relevant disciplines are also members of the Review Board (RB) at each UNT administration site which reviews students’ appeals with regard to the test items at the exam. One central Republican Committee to review appeals (RCRA) is established to ensure compliance with uniform requirements, 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 resolve disputes with regard to the exam questions as well as protect UNT participants’ rights for the period of UNT administration. The RCRA reviews the validity of appeal(s) to add scores to a certain examinee(s) submitted by the local RB and makes its’ final decision. In addition, the RCRA requests the RB to submit copies of the examinees’ answer sheets as well as confirmation of compliance with UNT procedures. The RCRA makes decision based on a majority of votes. In case the number of votes is equal, the vote of the RCRA’s chair will serve as the determining one. Local RBs are established at each UNT administration site as well. The RBs accept and review students’ appeals regarding the content of the exam questions and technical issues, make proposals to the RCRA regarding adding scores to a certain examinee, and informs the examinee on results of the RCRA’s decision. The minutes of the RCRA’s and RBs’ sittings are reflected in the minutes which are stored at the National Testing Center for one year. Members of the State Committee as well as parents whose children will be sitting for the UNT exam in the particular year cannot be the members of the RCRA or RBs. 22. There is a comprehensive technical report but with restricted circulation. The National Testing Center prepares test specifications which are approved by the Ministry of Education. These specifications are not available for general public. 23. Internal review or observers ensure the quality of the examination. 24. Inappropriate behaviors that diminish the credibility of the examination typically include Leakage of the content of an examination paper or part of a paper prior to the examination, impersonation when an individual other than the registered candidate takes the examination copying from other candidates, using unauthorized materials such as prepared answers and notes, collusion among candidates via mobile phones, passing of paper, or equivalent, intimidation of examination supervisors, markers or officials, and provision of external assistance via the supervisor, mobile phone etc. 25. The results are perceived as credible by some stakeholder groups. 26. All students may take the examination, regardless of background (e.g., gender, ethnic group), location (e.g., urban, rural), ability to pay (e.g., transportation, fees) or the like. 27. Limited information is available support the selection for this dimension. 28. Student names and results are public. Specifically, students’ UNT scores are published at each UNT administration site. The list of students who received the state education grants based on the UNT results are published in two major national newspapers “Kazakhstanskaya Pravda� (www.kazpravda.kz) and “Egemen Kazakhstan� (www.egemen.kz). The list is also published on the government website. 29. Students who do not perform well on the examination may opt for less selective schools/universities/tracks. According to the Rules on administering the UNT, each examinee may sit for the exam once. In case of failing the examination, the student will have the option to sit for the final general subject examinations at their secondary schools. The students will take the final subject examinations the same year they failed the UNT exam. These exams are designed and administered by the schools themselves (just like final exams) and the results are scored by the Commission on final examinations (the 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Commission) whose members consist of heads of education institutions and teachers in relevant subject areas. The content of examinations is approved by the district departments of education. Upon passing the examinations, the students receive the Diploma of General Secondary Education. These students are only allowed to apply to colleges (who grant post-secondary education) but not HEIs that year. In order to apply to HEIs next year, these students may take special comprehensive exams the following year which are administered by the HEIs and overseen by the Department of Higher and Post-Graduate Education at the Ministry of Education and Science of RK. 30. No systemic mechanisms are currently in place to monitor the consequences of the examination. The Center and the National Testing Center analyzes the results of the UNT. However, the analysis is usually limited to descriptive nature, and no studies on the impact or predictive validity take place. 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 KAZAKHSTAN National (or System-Level) Large-Scale Assessment (NLSA) 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this 1 irregular basis. been operating regularly. dimension. There is no policy document pertaining There is an informal or draft policy * There is a formal policy document that This option does not apply to this 2 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 3 dimension. the public. public. dimension. * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the 4 dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. 5 the NLSA or are indifferent to it. NLSA. NLSA. (CONTINUED) 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 6 NLSA. the NLSA. NLSA. dimension. * This option does not apply to this Funding covers some core NLSA Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. 7 and reporting. reporting. This option does not apply to this Funding does not cover research and This option does not apply to this * Funding covers research and 8 dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this 9 team. group of people. institution or unit. dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper 10 dimension. technical considerations. hamper technical considerations. technical considerations. This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this 11 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no 12 minimal issues. issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the opportunities to prepare individuals for 13 NLSA. NLSA. work on the NLSA. * 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this 14 curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely 15 questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. 16 measures what it is supposed to intended to measure. is intended to measure. measure. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality 17 the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 18 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in * 19 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. * There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality 20 documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA 21 NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. dimension. NLSA information is not used or is used This option does not apply to this NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and 22 assessment. technical characteristics of the technical characteristics of the assessment. assessment. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 23 monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of the NLSA. 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 National (or System-Level) Large Scale Assessment (NLSA): Development-level rating justifications 1. External Assessment of Learning Achievements (EALA) is aimed at assessing the quality of education services and the level of students’ mastery of secondary school curricula at the end of basic secondary education and higher education curricula in relevant disciplines. The main purposes of the EALA are to: monitor education quality at the system level; for school or educator accountability; and for policy design, evaluation, and decision making. EALA was administered for the first time on April 20 and 23, 2012. Ten percent of students in 9th grade from selected general secondary schools were assessed in four subjects (Kazakh language for schools with Kazakh language of instruction or Kazakh language for schools with Russian language of instruction, History of Kazakhstan, Algebra, and Chemistry). Starting in 2013, the Government will begin a transition to a 12�year school model. This transition is expected to be completed by 2020, at which point students will take EALA at grade 10 for further streaming into upper secondary education, (e.g. into humanities or technical fields), at grade 12 as a school leaving exam (to obtain a general secondary education diploma), and at the final year at the university. All three administrations will cover all students in the selected grade. Students will be tested in the Kazakh language (compulsory) and in three additional subjects that will be made known on the day of the test administration. 2. The Law of the Republic of Kazakhstan “On Education� authorized by the Decree of the President of the Republic of Kazakhstan (authorized on July 27, 2007 with the latest additions and amendments dated January 9, 2012) is a formal policy document that authorizes the EALA. 3. The policy document is publicly available on the websites of the Ministry of Education and other government agencies. 4. While there is no single stand�alone published large�scale assessment plan for the coming years or future assessment rounds, the law “On Education,� with the latest amendments and the State Program for Education Development 2011�2020, provide for the EALA as a form of independent monitoring of the quality of education. On November 7, 2011, the Ministry of Education approved an Implementation Plan on organization and implementation of EALA in education institutions in the Republic of Kazakhstan for 2011�2012 (the Plan) which specifies the activities involved in implementing the EALA and the deadlines by which the activities should be completed along with the responsible agencies. On April 6, 2012 the Ministry of Education also approved an Instruction on implementation of EALA in education institutions of the Republic of Kazakhstan which provides for detailed organizational and implementation arrangements for EALA in secondary and higher education institutions. This Instruction specifies the goal of EALA, grade levels tested, and the mandatory subject to be tested. The other three additional subjects will be annually selected and approved by the Ministry of Education. The Plan and the Instruction are publicly available documents. 5. Limited information is available support the selection for this dimension. 6. Regular (continuous and predictable) funding is allocated by the government. 7. Activities covered by the funding allocated for the EALA also include long� or medium�term planning of program milestones and staff training. 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 8. Funding does not cover research and development activities. 9. The National Testing Center is under the Ministry of Education and is responsible for test development and on�site administration of the EALA. The Division of External Examinations within the Committee for Quality Control in Education and Science under the Ministry of Education ("the Committee") is in charge of the policymaking regarding the national large�scale assessment program on behalf of the Ministry of Education. The National Testing Center reports to the Committee which also oversees that the EALA is implemented in compliance with the rules/implementation instructions. 10. The EALA was administered for the first time on April 20 and 23, 2012. Therefore, limited information is available support the selection for this dimension. 11. The NTC, which is responsible for carrying out the EALA, is accountable to the Committee for Quality Control in Education and Science, which oversees the implementation/administration of the EALA on behalf of the Ministry of Education. Both the NTC and the Committee for Quality Control in Education and Science are under the Ministry of Education. 12. There is an adequate number of full�time staff that is responsible for EALA activities. No issues have been identified with the performance of the human resources responsible for the EALA. 13. Non�university training courses/workshops on educational measurement and evaluation, and funding for attending international programs or courses/workshops on educational measurement and evaluation are available in Kazakhstan on an annual basis. The National Center for Education Statistics and Assessment (the Center) offers professional training courses in the area of education quality monitoring, management. Some of the training courses focus on education measurement (“New approaches to test development in the context of international experience�, “Managing the process of education measurement in education system�). The NTC also hosts various training workshops on the topic of educational measurement, often with participation of international experts. NTC also holds training workshops/courses annually in the areas of general theory of educational measurement, format of test items, test content criteria, which are led by Prof. Vadim Avanessov, a well respected Russian test development expert and the editor of the journal “Educational Measurement�. In 2007 the NTC became an IAEA member. The 38th IAEA Conference will be held in Kazakhstan in 2012. 14. The Law “On Education" specifies that the EALA is aimed at assessing the quality of education services and at determining the level of mastery in basic secondary and general secondary education programs of the compulsory education standards. 15. The EALA was administered for the first time on April 20 and 23, 2012. Therefore, limited information is available support the selection for this dimension. 16. The NTC is responsible for test development. The NTC’s website provides that development of test items is carried out by the team of test developers and the Experts Council approved by the Ministry of Education from qualified secondary school teachers, candidates and doctors of sciences, university professors etc. Experts’ review of test items is generally performed by secondary school teachers in specific disciplines. 42 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 17. Courses or workshops on the EALA are offered to teachers on a regular basis. Most courses provide teachers with relevant resources that they can use in their classrooms. NTC annually holds scientific and practical workshops to provide methodological assistance and improve the level of training of item writers and reviewers. These courses and workshops are mostly targeted at individuals involved in test development, item writing, and education measurement in general. Training workshops are provided largely to secondary school teachers involved in item writing or reviewing. 18. Special plans are made to ensure that the EALA is administered to students in hard�to�reach areas, and the assessment is offered in the language of instruction for almost all student groups. The EALA will be held at 154 sites across the country which are the same administration sites used to hold the Unified National Testing. This number of sites ensures that students from remote areas do not have to travel far to sit for the examination/assessment. EALA is administered in both the Kazakh and Russian languages which serve as the languages of instruction for most of the student groups in the country. 19. Mechanisms that ensure the quality of the EALA instrument include: all proctors or administrators are trained according to a protocol; there is a standardized manual for large�scale assessment administrators; a pilot is conducted before the main data collection takes place; all booklets are numbered; and external reviewers or observers are employed. A publically available list identifies unpermitted behaviors on test day (e.g. changing seats, use of calculators and mobile phones, cheating, leaving the room etc.). The pilot testing of test items for EALA was carried out by the NTC and the Department of preschool and secondary education of the Ministry of Education in December 2011. 20. Technical specifications which are prepared for the EALA in the form of a report, are not publically available. 21. The EALA was administered for the first time on April 20 and 23, 2012. Therefore, limited information is available support the selection for this dimension. 22. The EALA was administered for the first time on April 20 and 23, 2012. Therefore, limited information is available support the selection for this dimension. 23. Limited information is available support the selection for this dimension. 43 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 KAZAKHSTAN International Large-Scale Assessment (ILSA) 44 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. 2 the next 5 years. the next 5 years. There is no policy document that There is an informal or draft policy There is a formal policy document that * This option does not apply to this 3 addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA. This option does not apply to this The policy document is not available to The policy document is available to the * This option does not apply to this 4 dimension. the public. public. dimension. * ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, 5 ILSA. donors. discretion. decree or norm. * This option does not apply to this Funding covers some core activities of Funding covers all core activities of the This option does not apply to this 6 dimension. the ILSA. ILSA. dimension. * Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and 7 development activities. dimension. dimension. development activities. * (CONTINUED) 45 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. 8 activities. activities. activities. * This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this 9 dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the assessment. * This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, 10 effectively. with minimal issues. with no issues. * 46 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this 11 international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. The country/system offers no This option does not apply to this * The country/system offers some The country/system offers a wide range 12 opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. * This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition 13 team members only. to the country's/system's ILSA team * members. 47 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical The country/system met all technical published. standards to have its data presented standards required to have its data standards required to have its data beneath the main display of the presented in the main displays of the presented in the main displays of the * 14 international report or in an annex. international report. international report. The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new 15 new knowledge on ILSA. dimension. dimension. knowledge on ILSA. * ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely 16 the country/system. in the country/system. the country/system. disseminated in the country/system. Products to provide feedback to schools This option does not apply to this * Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are 17 not made available. sometimes made available. systematically made available. There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the * There is wide media coverage of the ILSA 18 results. ILSA results. ILSA results. results. * If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision 19 used to inform decision making in the the country/system. country/system. making in the country/system. country/system. * It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' 20 on students' achievement levels. achievement levels. * 48 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 International Large Scale Assessment (ILSA): Development-level rating justifications 1. Kazakhstan has participated in three international large-scale assessments: TIMSS 2007 (grade 4), TIMSS 2011 (grades 4 and 8), and PISA 2009. This rubric is completed primarily with regard to information from carrying out TIMSS 2011. 2. Kazakhstan has taken concrete steps to participate in PISA 2012, TIMSS 2015, and PIRLS 2016. 3. “State Program for Education Development of the Republic of Kazakhstan 2011-2020" is a formal document, authorized by the Decree of the President of the Republic of Kazakhstan № 1118 (Government of RK), December 7, 2010, that addresses participation in ILSA. 4. The policy document is publicly available at the Ministry of Education and other government websites. 5. Allocation of funding for ILSA activities is provided for by the government budget (divided into various budget programs and sub-programs), where implementation of international assessments is part of a certain budget program. 6. Funding covers international participation fees, implementation of the assessment exercise in the country (e.g., printing booklets, travel to schools), processing and analyzing data collected from implementation of the assessment exercise, reporting and disseminating the assessment results in the country/system, and attendance at international expert meetings for the assessment exercise. 7. Funding does not cover research and development activities. 8. The Director of the National Center for Education Statistics and Assessment under the auspices of the Ministry of Education and Science of RK (the Center, www.quality.edu.kz, www.naric.kz) (Ms. Aigul Kultumanova) is the national/system coordinator responsible for the PISA and TIMSS international assessments in Kazakhstan. The team at the Center is responsible for carrying out international assessments in the country. The Center houses 4 institutes, one of which is the Institute for Educational Measurement , which monitors the status and education development trends in Kazakhstan, develops proposals/recommendations on improving mechanisms for education quality management in the country. Implementation of the international assessment in the regions of Kazakhstan is carried out with direct technical assistance of the sub-national departments of education. In each region a Regional Coordinator was assigned who was responsible for organization and implementation of the assessment. As reported by the Center’s staff, the Center had recently been renamed as well as had undergone a re-organization which would result in the upcoming change of the Center’s organizational structure. 9. The national coordinator is fluent in the language of the assessment. 49 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 10. Some of the team members have the necessary training to carry out the required assessment activities effectively as well as previous experience working on ILSAs. Specifically, the team at the Laboratory for International Studies (the Laboratory) at the Center responsible for carrying out the ILSAs consists of a team of 11 employees, four of which have been working at the Laboratory since 2010. While team members have prior working experience in education, teaching, foreign language specialists, most of them do not have extensive experience/training in the area of international assessments. Except for one individual that has prior experience in UNT exam implementation on a regional level, the rest of the staff do not seem to have any training and/or experience in other types of assessment. During times of overlap of international assessments, there can be a shortage of human resources, in which case technical work, such as data entry and logistical arrangements are outsourced. Data collection/processing, any type of data analysis and interpretation of findings are not outsourced. There have been delays in the printing of the test booklets, particularly during the overlap in the implementation of TIMSS and PISA, as the Center's editorial- and-publishing service which, among others, produces print outs of test booklets, does not have sufficient capacity to accommodate a large volume of printing. Occasionally, the printers failed, causing delays as well. 11. Team members have attended some of the meetings. Most of the training opportunities (because of the limited funding) are limited to 1 or 2 staff members, usually including the Director. The team participated in each National Research Coordinators meeting arranged by the IEA from 2009-2011. Participation at the 2011 NRC meeting was limited to the Director of the Laboratory for International Studies only. In 2010, the Center’s team also attended training seminar on international data management (as it pertained to the upcoming TIMSS-2011 assessment) and the use of the supporting software arranged by the IEA-DPC in Hamburg, Germany. The funding for any staff training opportunities is very limited, and participation of all staff members/or participation in all international meetings related to international assessment is not possible. 12. There are no university/on-line courses on the topic of international assessments offered in the country, i.e. there are no system-level regular learning opportunities related to international assessments in Kazakhstan. However, before upcoming assessment studies, the Center holds training workshops for the Regional Coordinators and other individuals responsible for the technical implementation of the assessment in the regions. The National Testing Center also hosts various training workshops on the topic of educational measurement, often with participation of international experts. 13. Opportunities to learn about international assessments are offered to individuals working directly on the specific international assessment exercise. 14. Kazakhstan met all technical standards required to have its data presented in the main displays of the international report. 15. Kazakhstan has not contributed new knowledge on ILSA. 16. Country-specific results from the international assessment (TIMSS 2007, as TIMSS 2011 results will be disseminated in December 2012) are disseminated in a number of ways. The Center publishes an annual National Report on the Status of Education in the Republic of Kazakhstan (the National Report). This report is generally available for download at the Ministry of Education’s website and the website of the Center. The 2009 and 2010 National Reports include a chapter on TIMSS 2007 assessment results. The current link for the National Report 2009 provided for download at the Ministry’s web-site is not active. The National Report 2010 is available for download at the Ministry’s website. In 2009, the Center published the National Report on TIMSS 2007results. The Center's website is currently under construction based on their recent restructuring from the National Center for Education Quality Assessment into the National Center for Education Statistics and Assessment, therefore the report is not currently available for download. Copies of the TIMSS 2007 National 50 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Reports were shared with departments of education in the regions and local executive bodies. Additionally, TIMSS 2007 results were communicated through a press release and were also covered on television, radio and newspapers. 17. Limited information is available on how ILSA results are fed back to schools and educators. The Center normally works with sub-national departments of education to disseminate results. The departments in turn work with schools and educators on a local level. The Pavlodar oblast, for example, provides information on its department of education website, briefly reporting on the significance of international assessments for Kazakhstan, general aspects of the TIMSS-2011 study, sample information etc. The website also indicates that it shared the sample of TIMSS items with local schools for their familiarization with the format of the examination. 18. International assessment results have been discussed in major newspapers and information agencies in the country and press-releases have been published, among other places, on the Ministry of Education and Government of Kazakhstan websites. It may be said that the results of TIMSS 2007 results have been adequately covered by the media. For example, Kazakhstan Today, a major information agency, published two press-releases on the results. Kazakhstan law internet portal, which is also a source of the major news in Kazakhstan, published on Kazakhstan’s relatively high results on TIMSS. The Ministry of Education and the Government of RK’s websites published press releases which briefly discussed the results of TIMSS 2007. 19. Limited information is available on whether and how ILSA results are used to inform policy-making to improve education quality. While there are recommendations on how assessment results can be used to inform division making in Kazakhstan, assessment results (specifically from PISA 2009), have been the focus of discussions on student learning in Kazakhstan. The President of RK, for example, instructed the government to prepare a 5-year National work plan on functional literacy development among school children in Kazakhstan which is currently under preparation by the Ministry of Education and other appointed agencies. This policy initiative is largely based on Kazakhstan’s disappointing results PISA 2009. Assessment results have also been used to continue dialogue on student learning in the country. In a press release published on the Ministry of Education’s website in February 2012, the Deputy Prime Minister and the Minister of Education mention that Kazakh pupils do not have the skills to apply acquired knowledge to real world situations based on the results of PISA 2009 assessment. The Minister of Education emphasized that one of the main problems lies with the lack of good teachers and the low status of teachers in the country. The existing State Program for Education Development 2011-2020 provides for policy measures targeted at improving teacher education programs and raising the status of teachers in Kazakhstan. 20. The Center is in need of in-house technical expertise to carry out an in-depth analysis of assessment result 51 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Acknowledgements This report was prepared by The World Bank SABER- Student Assessment team in collaboration with Scherezad Latif, World Bank Senior Education Specialist and Task Team Leader for education projects in Kazakhstan, Chelsea Coffin, World Bank Education Specialist, HDNED, and Tatiana Li, World Bank Education Specialist, ECAEDU. References Clarke, M. 2012. “What Matters Most for Student Assessment Systems: A Framework Paper.� READ/SABER Working Paper Series. Washington, DC: World Bank. World Bank. 2013. World Bank Development Indicators: Kazakhstan Country Indicator Data. Washington, DC: World Bank. Data retrieved from http://databank.worldbank.org/data on March 12, 2013. ———. 2012. “Student Assessment System Policy Note: Kazakhstan 2012.� Unpublished report. Washington, DC: World Bank. 52 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS KAZAKHSTAN ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK 2 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS