67663 The World Bank Human Development Network - Education System Assessment and Benchmarking for Education Results (SABER) SAINT KITTS AND NEVIS Education Management Information System (EMIS) COUNTRY REPORT Emilio Porta, Jennifer Klein, Gustavo Arcia and Harriet Nannyonjo February 2012 Acknowledgements This report was prepared by a team led by Emilio Porta, Senior Education Specialist at the Human Development Network/Education at the World Bank; and consisting of Gustavo Arcia, Consultant to the Human Development Network/Education of the World Bank and Senior Economist at Analítica LLC in Miami, Florida; Jennifer Klein, Consultant to the Human Development Network/Education at the World Bank, and Harriet Nannyonjo, Senior Education Specialist, LCSHE, World Bank. The report was prepared under the guidance of Elizabeth King, Robin Horn and Chingboon Lee. The views expressed here are those of the authors and should not be attributed to the World Bank Group. All data contained in this report is the result of collaboration between the authors, the Organization of Eastern Caribbean States, and participants in the benchmarking exercise. All errors are our own. This benchmarking study arose from an active partnership between the Education Reform Unit of the Organization of Eastern Caribbean States (OECS) and the World Bank. The benchmarking exercise was done during an OECS workshop conducted in Castries, St. Lucia, from January 23 to January 28, 2011, with the participation of government officials from Antigua & Barbuda, the Commonwealth of Dominica, Grenada, St. Kitts & Nevis, St. Lucia, and St. Vincent & the Grenadines. A delegate from Montserrat also attended as an observer. The workshop and benchmarking exercise were done under the invaluable leadership of Marcellus Albertin, Head of the Education Reform Unit (OERU) at the OECS. His unflagging support, enthusiasm, and institutional supervision were fundamental for the cooperation of all participants and for the success of the workshop. To him we owe a great deal of gratitude. We would like to thank the OERU staff that helped us with workshop logistics, especially Emma Mc Farlane- Jouavel and Beverly Pierre. We would also like to thank the workshop participants: Doristeen Etinoff, Priscilla Nicholas, and Patricia George from Antigua & Barbuda; Ted Serrant, Robert Guiste, and Weeferly Jules from Dominica; Pauleen Finlay, Michelle Peters, and Imi Chitterman from Grenada; Gregory Julius from Monserrat; Quinton Morton, Ian Gregory, and Laurence Richards from St. Kitts & Nevis; Kendall Khodra, Nathalie Elliott, Sisera Simon, Evariste John, and Valerie Leon from St. Lucia; Dixton Findlay, Keith Thomas, and Junior Jack from St. Vicent & Grenadines; Darrel Montrope, Jacqueline Massiah, Sean Mathurin, and Loverly Anthony- Charles from the OECS. Abbreviations EMIS Education Management Information System MOE Ministry of Education OECD Organization for Economic Cooperation and Development OECS Organization of Eastern Caribbean States SABER System Assessment and Benchmarking for Education Results SEAT SABER EMIS Assessment Tool UIS UNESCO Institute for Statistics UNESCO United Nations Educational, Scientific, Cultural Organization SAINT KITTS AND NEVIS ESTABLISHED  Aspect of Data Quality Benchmark Prerequisites of Quality Established ¤¤¤¢ Assurances of Integrity Emerging ¤¤¢¢ Methodological Soundness Established ¤¤¤¢ Accuracy and Reliability Established ¤¤¤¢ Serviceability Established ¤¤¤¢ Accessibility Established ¤¤¤¢ 1 BACKGROUND Education Data in St. Kitts and Nevis The Education Management Information System (EMIS) in St. Kitts and Nevis is part of the Education Planning Division of the Ministry of Education. It functions in collaboration with the Chief Education Officer and other education officers. Four key objectives of St. Kitts’ EMIS have been established: Ü Collect and analyze educational data. Ü Inform policymakers. Ü Provide feedback to stakeholders. Ü Monitor and support schools. FACILITIES AND EQUIPMENT. All schools are equipped with administrative computers and broadband Internet access, which allow for the internet-based exchange of information and email-based communication. EMIS data are stored the EMIS office and are backed up both electronically and in hard copy. Occasional issues with the server have been noted as a challenge by EMIS staff. EMIS STAFF. St. Kitts’ EMIS staff includes an EMIS Director, secretary, and Education Officers. Also, each school selects an Information and Communication Technology (ICT) Champion, who is the link between the EMIS staff and the school. EMIS DATA. Data on students and teachers are collected from schools and other educational institutions, and additional teacher data is collected from the Human Resources Unit. The Ministry of Education’s Project Management Unit and schools supply infrastructure data, and budget data is collected from schools and the Statistics Unit. DATA COLLECTION. Data collection begins in September with the submission of questionnaires to the 24 public primary schools, nine private primary schools, eight public secondary schools, and three private secondary schools. The annual questionnaires are collected by October 31, but additional student and teacher attendance forms are submitted each term. DATA PROCESSING. The Ministry of Education EMIS department aggregates the source data received from various sources. The Statistical Officer verifies accuracy and consistency of the data, and errors are communicated to the school/institution. The Statistical Officer also designs the output tables. PUBLICATIONS. At the end of the school year in August, St. Kitts strives to publish a statistical bulletin that presents EMIS data and the financial expenditure of the Ministry of Education. Also, the EMIS provides annual statistics upon request to regional and international affiliates including the OECS Educational Research Unit (OERU) and UNESCO. 2 The EMIS in St. Kitts & Nevis ESTABLISHED:  In January 2011, St. Kitts and Nevis’ EMIS was assessed using the SABER-EMIS Assessment Tool (SEAT) and overall the EMIS was categorized as ESTABLISHED (0.65). Among the six Organisation of Eastern Caribbean States (OECS) countries, St. Kitts tied with Dominica for the highest overall score (Table 1). St. Kitts outperformed the OECS average Figure 1. SABER EMIS Scores in the OECS Countries on most of the SEAT’s Aspects of Quality except two – Methodological Soundness and Assurances of Integrity. St. Kitts’ Methodological Soundness score was only slightly below average due to one EMERGING score on the scope of statistics, but St. Kitts had many EMERGING sub-component scores on the Assurances of Integrity that resulted in the lowest average score of all the OECS countries. St. Kitts was the highest of the OECS countries on two aspects – Accuracy & Reliability and Serviceability – but neither of the two Aspects had MATURE average scores. The next sections of this country report will analyze St. Kitts’ performance on the sub-components of each Aspect of Quality in order to present a detailed portrait of the strengths and weaknesses of St. Kitts’ EMIS and many concrete actions that the country can take to improve education data quality. Table 1. SABER EMIS Scores in the OECS Countries (2011) OECS Dominica Antigua Grenada St. Kitts St. Vincent St. Lucia Average Pre-Requisites 0.70 0.52 0.68 0.66 0.45 0.64 0.61 of Quality Assurances of 0.58 0.53 0.61 0.44 0.50 0.64 0.55 Integrity Methodological 0.83 0.50 0.67 0.67 0.83 0.67 0.69 Soundness Accuracy and 0.70 0.48 0.58 0.75 0.53 0.58 0.60 Reliability Serviceability 0.61 0.29 0.50 0.79 0.43 0.68 0.55 Accessibility 0.47 0.47 0.69 0.61 0.36 0.56 0.53 Overall 0.65 0.46 0.62 0.65 0.52 0.63 0.59 Latent Emerging Established Mature 0 – 0.3 0.31 - 0.59 0.6 - 0.79 0.8 - 1 3 PREREQUISITES OF QUALITY ESTABLISHED:  Figure 2. Prerequisites of Quality St. Kitts has ESTABLISHED (0.66) the Prerequisites of Quality (Figure 2) necessary to support an EMIS and was ESTABLISHED above the OECS average score, but still has a few areas in    need of improvement. Processes are in place to ensure that human, physical, and data resources are efficiently managed (Table 2, 0.6) but current EMIS staffing and training levels are inadequate (0.5). Quality procedures are in place and enforced by management (0.8), but there are no reviews by external bodies or user surveys to monitor the quality of data (0.9). The legal framework specifying responsibilities for collecting and disseminating data are not specific to the education sector (0.1). An informal agreement exists for data sharing and coordination among most educational levels and institutions, but it is not fully implemented; there is no agreement for data sharing or coordination with private universities (0.2). Statistical reporting is encouraged through an informal agreement, but there are no formal legal mandates or penalties for institutions that do not report data (0.4). St. Kitts could improve the Prerequisites of Quality by formalizing informal agreements, establishing the formal legal framework for the EMIS, and hiring and training additional EMIS staff. St. Kitts & OECS Table 2. Prerequisites of Quality: Subcomponents Benchmark Nevis Average Responsibility for collecting and disseminating education data is Established 0.1 0.75 0.75 clearly specified  Emerging 0.2 Data sharing and coordination among different agencies are adequate 0.25 0.50  Individual/personal data are kept confidential and used for statistical Established 0.3 0.75 0.79 purposes only  Statistical reporting is ensured through legal mandate and/or Emerging 0.4 0.25 0.58 measures to encourage response  Staff, facilities, computing resources, and financing are Emerging 0.5 0.50 0.63 commensurate with the activities  Processes and procedures are in place to ensure that resources are Mature 0.6 1.00 0.63 used efficiently  Education statistics meet user needs and those needs are monitored Established 0.7 0.75 0.75 continuously  Mature 0.8 Processes are in place to focus on quality 1.00 0.63  Emerging 0.9 Processes are in place to monitor the quality of data processes 0.50 0.33  Processes are in place to deal with quality considerations in planning Established 0.10 0.75 0.58 the stat program  Mechanisms exist for addressing new and emerging data Established 0.11 0.75 0.54 requirements  4 ASSURANCES OF INTEGRITY EMERGING:  Assurances of Integrity was St. Kitts’ only EMERGING Figure 3. Assurances of Integrity in the OECS average score (0.44) and the only instance where St. Kitts scored drastically below the OECS average (0.55). A closer look at the sub-components of this Aspect reveals a wide range of scores from LATENT to MATURE. EMERGING Only informal mechanisms protect the professional    independence of the data producing institution, which can ensure that statistics are produced on an impartial basis (Table 3, 1.1). More formal arrangements could be developed to improve the score on this sub-component. Choices of data sources are technically justified and staff are encouraged to enforce technical criteria (1.3). The terms and conditions under which statistics are collected, processed, and disseminated are difficult to find but made available on request (1.5). No notice is given on changes in methodology, source data, or statistical techniques (1.8). St. Kitts’ earned its only MATURE score by having well- known professional guidelines for staff behavior (1.1). Even though professional credentials are already considered for recruitment and promotion (1.2), professionalism could be furthered by encouraging staff to publish and having a peer review process in place. St. Kitts & OECS Table 3. Assurances of Integrity: Subcomponents Benchmark Nevis Average Emerging 1.1 Statistics are produced on an impartial basis 0.25 0.38  Emerging 1.2 Professionalism of staff is actively promoted 0.50 0.42  Choices of data sources and statistical techniques are made solely by Established 1.3 0.75 0.83 statistical considerations  Agency is entitled to comment on erroneous interpretation and misuse Emerging 1.4 0.50 0.58 of statistics  Emerging 1.5 Terms and conditions are available to the public 0.25 0.33  Public is aware of internal governmental access to statistics prior to Emerging 1.6 0.50 0.38 their release  Emerging 1.7 Products of education statics agency are clearly identified 0.25 0.50  Advanced notice is given of major changes in methodology, source Latent 1.8 0.00 0.71 data, and statistical techniques  Guidelines for staff behavior are in place and are well known to the Mature 1.9 1.00 0.83 staff  5 METHODOLOGICAL SOUNDNESS ESTABLISHED:  In terms of Methodological Soundness, St. Figure 4. Methodological Soundness in the Kitts’s EMIS is ESTABLISHED (0.67). St. Kitts OECS countries scored just below the OECS average (0.69) and had the same score as both Grenada and St. Lucia (Figure 4). ESTABLISHED St. Kitts earned one MATURE score on this    Aspect of Quality due to its use of the internationally accepted standards and guidelines for structure, concepts and definitions established by the UNESCO Institute for Statistics (UIS) and the OECS Education Reform Unit (OERU) (Table 4, 2.1). St. Kitts also follows the International Standard Classification of Education (ISCED) in all education sector data except expenditure data (2.3). Expanding the use of ISCED to expenditure data would ensure complete consistency with ISCED and improve St. Kitts’s score on this subcomponent. Currently, St. Kitts’s EMIS produces around 69 percent of UIS indicators annually, which results in a EMERGING benchmark on the scope of statistics sub-component (2.2). Expanding the scope of statistics produced to 100 percent of UIS and OECD indicators is ideal and can enable additional domestic, regional, and international education policy analysis. St. Kitts & OECS Table 4. Methodological Soundness: Subcomponents Benchmark Nevis Average Overall structure, concepts and definitions follow regionally and Mature 2.1 internationally accepted standards, guidelines, and good 1.00 0.83  practices Scope is in accordance with international standards, guidelines, Emerging 2.2 0.25 0.42 or good practices  Classification systems are consistent with international Established 2.3 0.75 0.83 standards, guidelines, or good practices  6 ACCURACY AND RELIABILITY ESTABLISHED:  Figure 5. Accuracy and Reliability The Accuracy and Reliability of St. Kitts’s EMIS statistics is ESTABLISHED (0.75) (Figure 5). St. Kitts’ ESTABLISHED score was the highest of the OECS countries, and on four    of 10 sub-components, St. Kitts’s EMIS was MATURE (Table 5). Data compilation in St. Kitts employs sounds statistical techniques including appropriate sample sizes for survey data and updated school registries for census data (3.5). Statistical discrepancies in intermediate data are always assessed and investigated (3.8). There are systematic processes in place for monitoring errors and omissions and the results are made public (3.9). Also studies and analyses of revisions are carried out routinely. A feedback loop is implemented based on the studies to inform the process and the findings are made public (3.10). In order to improve, St. Kitts could 1) collect data on school characteristics and education demand (3.1), 2) develop procedures to update, standardize, and properly reference source data (3.2), 3) document information on sampling errors and imputed data (3.4); 4) always validate intermediate results against other information (3.7) and finally 5) ensure that education data are provided within six months after the end of the school year to other source providers (3.3). St. Kitts & OECS Table 5. Accuracy and Reliability: Subcomponents Benchmark Nevis Average Source data are obtained from comprehensive data collection that Emerging 3.1 0.50 0.58 takes into account country-specific conditions  Data are reasonably confined to the definitions, scope, classifications, Emerging 3.2 0.50 0.50 and time of recording required  Established 3.3 Source data are timely (6 months after event) 0.75 0.46  Other data sources, such as censuses, surveys, and administrative Emerging 3.4 0.50 0.42 records, are routinely assessed  Data compilation employs sound statistical techniques to deal with Mature 3.5 1.00 0.79 data sources  Other statistical procedures (data editing, transformations, and Established 3.6 0.75 0.63 analysis) employ sound statistical techniques  Intermediate results are validated against other information where Emerging 3.7 0.50 0.67 applicable  Statistical discrepancies in intermediate data are assessed and Mature 3.8 1.00 0.92 investigated  Statistical discrepancies and other potential indicators or problems in Mature 3.9 1.00 0.71 statistical outputs are investigated  Studies and analyses of revisions are carried out routinely and used Mature 3.10 1.00 0.33 internally to inform the processes  7 SERVICEABILITY ESTABLISHED:  The Serviceability of St. Kitts’s EMIS data is Figure 6. Serviceability in the OECS ESTABLISHED (0.79) and is far above the OECS ESTABLISHED average (0.55). St. Kitts’s lowest score on any sub-    component was 0.50, which indicates that a strong foundation for Serviceability is currently in place. St. Kitts met the MATURE benchmark for Periodicity by producing an annual census of enrolments, teachers, schools, and financial data (Table 6, 4.1), but the timeliness of releasing the data could be improved: Currently administrative census data are available six to 12 months after the initiation of the school year when ideally this data should be released within two months (4.2). Time series data are available for five to 10 years, and there are procedures in place for the revision of time series data (4.4). Crosschecking is done regularly but consistency checking is done only for administrative census data (4.3). When school- reported data was checked for consistency against household survey data, the difference between the two sets of data was less than five percentage points, thus earning St. Kitts a MATURE score on sub- component 4.5. St. Kitts could improve its Serviceability by publishing administrative data within two months after the initiation of the school year, increasing the availability of time series data to more than 10 years, and strengthening systems for revisions. St. Kitts & OECS Table 6. Serviceability: Subcomponents Benchmark Nevis Average Mature 4.1 Periodicity follows dissemination standards 1.00 0.96  Emerging 4.2 Timeliness follows international dissemination standards 0.50 0.63  Established 4.3 Statistics are consistent within the dataset 0.75 0.71  Statistics are consistent or reconcilable over a reasonable Established 4.4 0.75 0.54 period of time  Statistics are consistent or reconcilable with those obtained Mature 4.5 1.00 0.33 through other data sources and/or statistical frameworks  Emerging 4.6 Revisions follow a regular and transparent schedule 0.50 0.21  Mature 4.7 Preliminary and/or revised data are clearly identified 1.00 0.46  8 ACCESSIBILITY ESTABLISHED:  St. Kitts earned the second highest Accessibility score of the OECS countries (0.61/ESTABLISHED), but St. Kitts’ sub-component scores ranged from LATENT to MATURE. St. Figure 7. Accessibility in the OECS Kitts had three MATURE scores: 1) EMIS statistics are clearly presented with disaggregation and underlying data for ESTABLISHED charts (Table 7, 5.1); 2) data are released to all users at the    same time (5.4) and 3) a data catalog is available so users can request data according to their needs (5.7). Data are available electronically (5.1), but data are not released on a pre-announced schedule (5.3) and yearbooks are only printed upon request (5.2). Metadata are documented and updated, but only available upon request (5.6). The release of non-published data and non-confidential data is discretionary (5.5) and no catalogs of publications and other services are available (5.9). Most statistical releases identify a contact person in case of required assistance and assistance is monitored (5.8). Accessibility is one of the key missions of an EMIS because it creates and maintains the public image of the EMIS and enables greater accountability. St. Kitts could develop a more accessible EMIS by establishing and pre-announcing the schedule for data releases, offering improved assistance for users, and creating a catalog of publications and other services. St. Kitts & OECS Table 7. Accessibility: Subcomponents Benchmark Nevis Average Statistics are presented to facilitate proper interpretation and Mature 5.1 1.00 0.96 comparisons (layout, clarity of texts, tables, and charts)  Established 5.2 Dissemination media and format are adequate 0.75 0.54  Emerging 5.3 Statistics are released on a pre-announced schedule 0.25 0.38  Mature 5.4 Statistics are made available to all users at the same time 1.00 0.79  Emerging 5.5 Statistics not routinely disseminated are made available upon request 0.50 0.75  Documentation on concepts, scope, classifications, basis of recording, data sources, and statistical techniques is available, and differences Established 5.6 0.75 0.58 from internationally accepted standards, guidelines, or good practices  are annotated Mature 5.7 Levels of detail are adapted to the needs of the intended users 1.00 0.38  Emerging 5.8 Contact points for each subject field are publicized 0.25 0.38  Catalogs of publications and other services, including information on Latent 5.9 0.00 0.00 any charges, are widely available  9