32759 No. 0529 Social Protection Discussion Paper Series Targeting Social Spending To The Poor With Proxy­Means Testing: Colombia's SISBEN System Tarsicio Castañeda June 2005 Social Protection Unit Human Development Network The World Bank Social Protection Discussion Papers are not formal publications of the World Bank. They present preliminary and unpolished results of analysis that are circulated to encourage discussion and comment; citation and the use of such a paper should take account of its provisional character. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. For free copies of this paper, please contact the Social Protection Advisory Service, The World Bank, 1818 H Street, N.W., Washington, D.C. 20433 USA. Telephone: (202) 458-5267, Fax: (202) 614-0471, E-mail: socialprotection@worldbank.org. Or visit the Social Protection website at http://www.worldbank.org/sp. Social Safety Net Primer Series Targeting Social Spending To The Poor With Proxy­Means Testing: Colombia's SISBEN System Tarsicio Castañeda* June 2005** The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. *Report prepared for the World Bank under the supervision of Kathy Lindert. Research assistance was provided by Luisa Fernandez. I am most thankful to Kathy Lindert for her detailed comments and suggestions made to an initial draft of the report. Findings and opinions are those of the author and not of the World Bank. Comments to: tcastaneda@tutopia.com. **This study was completed in June 2003. Social Safety Net Primer Series The World Bank Social Safety Nets Primer is intended to provide a practical resource for those engaged in the design and implementation of safety net programs around the world. Readers will find information on good practices for a variety of types of interventions, country contexts, themes and target groups, as well as current thinking of specialists and practitioners on the role of social safety nets in the broader development agenda. Primer papers are designed to reflect a high standard of quality as well as a degree of consensus among the World Bank safety nets team and general practitioners on good practice and policy. Primer topics are initially reviewed by a steering committee composed of both World Bank and outside specialists, and draft papers are subject to peer review for quality control. Yet the format of the series is flexible enough to reflect important developments in the field in a timely fashion. The primer series contributes to the teaching materials covered in the annual Social Safety Nets course offered in Washington, DC as well as various other Bank-sponsored courses. The Social Safety Nets Primer and the annual course are jointly supported by the Social Protection unit of the Human Development Network and by the World Bank Institute. The World Bank Institute also offers customized regional courses through Distance Learning on a regular basis. For more information on the primer paper series and papers on other safety nets topics, please contact the Social Protection Advisory Service; telephone (202) 458-5267; fax (202) 614-0471; email: socialprotection@worldbank.org. Copies of related safety nets papers, including the Social Safety Nets Primer series, are available in electronic form at www.worldbank.org/safetynets. The website also contains translated versions of the papers as they become available. An ambitious translation plan is underway (especially for Spanish and French, some in Russian). For more information about WBI courses on social safety nets, please visit the website www.worldbank.org/wbi/socialsafetynets. Papers in the Safety Nets Primer as of June 2005 Theme Author Program Interventions Cash Transfers Tabor, Steve Cash Transfers Benefits Lafaurie and Valasquez Community-based Health Insurance Tabor, Steve Conditional Cash Transfers Rawlings, Laura Fee Waivers in Health Bitran and Giedion Fee Waivers in Housing Katsura and Romanik Food Related Programs Rogers and Coates Micro Credit and Informal Insurance Sharma and Morduch Mitigating Social Risks Tesliuc, Emil Price and Tax Subsidies Alderman, Harold Public Works Subbarao, Kalanidhi Cross-cutting Issues Evaluation Blomquist, John Gender Ezemenari, Chaudhury and Owens Institutions de Neubourg, Chris Political Economy Aspects of Targeting Pritchett, Lant Public Attitudes and Political Economy Graham, Carol Safety Nets for Poverty Reduction Ravillion, Martin Targeting Coady, Grosh and Hoddinott Targeting: Lessons from LAC ­ Overview Lindert et all Targeting in Brazil Lindert and Brière Targeting in Chile (Spanish) Larrañga, Osvaldo Targeting in Colombia Castañeda, Tarsicio Targeting in Costa Rica (Spanish) Viguez, Roxana Targeting in Mexico (Spanish) Orozco and Hubert Testing Vietnam's Public Safety Nets van de Walle, Dominique Country Setting/Target Group Poverty and Aging in Africa Subbarao, Schwartz and Kakwani Transition Economies Fox, Louise Very Low Income Countries Smith and Subbarao Special Vulnerable Group Disability Mitra, Sophie LIST OF ACRONYMS AIDS Acquired Immune Deficiency Syndrome ARS Administrators del Régimen Subsidiado - Subsidized Insurance Companies CCT Conditional Cash Transfer CDB Central Data Base CONPES Document approved by the Council of Ministers and President DANE Departamento Nacional de Estadística - National Statistical Department DNP Departamento Nacional de Planeación ­ National Planning Department ENCV Encuesta Nacional de Calidad de Vida ­ Survey of Living Conditions ESE Estratificación Socioeconómica ­ Socio-Economic Stratification FOSYGA Fondo de Solidaridad y Garantía - Social Solidarity Fund GDP Gross Domestic Product ICBF Instituto Colombiano de Bienestar Familiar - Colombian Institute of Family Welfare ID Identification Number IDB Inter- American Development Bank LP Poverty Line MOH Ministry of Health MSP Ministry of Social Protection NBI Necesidades Basicas Insatisfechas - Unsatisfied Basic Needs. NGO Non ­ Governmental Organization POSS Plan Obligatorio de Salud Subsidiado ­ Subsidized Health Insurance Package PRINQUAL Qualitative Principal Components RAS Red de Apoyo Social ­ Social Safety Net SENA Servicio Nacional de Aprendizaje - National Training Institute SHIR Subsidized Health Insurance Regime SISBEN Sistema de Selección de Beneficiarios de Programas Sociales - System for Selecting Beneficiaries of Social Programs TA Technical Assistance TC Technical Committee UDS Unidad de Desarrollo Social ­ Social Development Unit UNDP United Nations Development Program WB World Bank Preface This case study is part of six Country Case Study Reports that were commissioned in 2003 by the World Bank specifically for the purposes of a summary report on the design and implementation of household targeting systems in the following countries: Chile, Colombia, Costa Rica, Mexico, Brazil and the United States. Research findings and earlier drafts of the report were presented at numerous workshops and seminars (two in Brazil in November 2003; two at the World Bank in Washington in November 2003 and January 2005; and one at the Second International Workshop of Conditional Cash Transfers in Sao Paulo in April 2004). The final version of the report as well as the other country case studies have been published as Social Protection Discussion Paper No. 0526 to 0532 and can be found at www.worldbank.org/safetynets. Abstract While targeting can effectively channel resources to the poor, implementation details matter tremendously to distributive outcomes. Several key factors affect performance, including: data collection processes; information management; household assessment mechanisms; institutional arrangements; and monitoring and oversight mechanisms. This report conducts an in-depth assessment of key design and implementation factors and their potential impact on outcomes for the household targeting system SISBEN used in Colombia to target social programs to the poor and vulnerable. Table of Contents Introduction....................................................................................................................... 1 Rationale For Targeting and Choice of Instrument in Columbia................................ 3 The SISBEN: Design and Implentation Features .......................................................... 6 Implementation of SISBEN............................................................................................ 17 Uses of SISBEN For Program Eligibility...................................................................... 23 Evaluations of Effectiveness/Outcomes of SISBEN..................................................... 32 Recent Update of System................................................................................................ 36 Summary and Recommendations.................................................................................. 39 References........................................................................................................................ 42 Annex 1. Legal Directives for Targeting and/or Use of SISBEN............................... 43 Annex 2. Summary Content of SISBEN Operating Manuals.................................... 44 Annex 3. Variables and Weights of Statistical Principal Component Model........... 45 Boxes 1. Use of SISBEN ............................................................................................................. 24 2. The Implementation and Use of SISBEN in the City of Bogotá.................................. 27 Tables 1. Roles and Repsonsibilities of Different Government Levels Design and Application of SISBEN ........................................................................... 7 2. Variables Included in SISBEN Questionnaire................................................................ 9 3. People Registered in SISBEN by Urban and Rural Area, 1993-2000.......................... 17 4. Distribution of SISBEN Population by SISBEN Level, 1993-2000 ............................ 17 5. Main Programs Using SISBEN for Selcting Beneficiaries, 2002-2003 ....................... 23 6. Evolution of Implementation of SISBEN and Affliation to the Subsidized Health Insurance Regime (SHIR) 1996-2002 .................................. 24 7. Uses of SISBEN by Muncipalities, 1994-2000 ............................................................ 27 8. Approximate Total Costs of SISBEN Registry in Bogotá, 2003.................................. 28 9. Estimated Annual Costs of SISBEN Registry in Bogotá, 2003................................... 28 10. Secretaries and Main Programs Using SISBEN to Select Beneficiaries, 2002......... 28 11. SISBEN and Other Targeting Criteria Used in Main Programs, 2002...................... 30 12. Shares of Subsidies Received by Deciles of Income Distribution 1997..................... 33 13.Share of Subsidies Received by the Bottom 20% and 40% of the Colombian Population, 1997 (%)............................................................................34 14. Approximate National Costs of SISBEN Registry, 2003........................................... 35 15. Estimated Annual Costs of SISBEN Application Nationally, 2002........................... 35 16. Variables Included in SISBEN Index and Questionnaire for Urban and Rural Areas (2003).................................................................................. 37 Targeting Social Spending To The Poor With Proxy­Means Testing: Colombia's SISBEN System Tarsicio Castañeda 1. Introduction The late 1980s and beginning of the 1990s marked the beginning of attempts to target social spending to the poor in most of Latin America countries. This resulted from fiscal realities, as most countries were faced with acute fiscal constraints, and from policy concerns, as policy makers and academicians saw targeting as one instrument to achieve universal access to services, rather than as a threat (a common view in the past) to achieving such an objective. Colombia was not the exception, and the 1991 Constitution mandated that social spending be targeted to the poor--people with unsatisfied basic needs (NBI in Spanish). Colombia has used two methods for targeting social spending. The first is a geographical targeting instrument, known as the Socio-economic Stratification (Estratificacion Socio-Económica or ESE), which is based on assessment of the outside characteristics of neighborhoods and dwellings. It is used to target subsidies for potable water, electricity and a variety of other small subsidies, by central and local governments. The second is a proxy-means testing instrument, known as System for Selecting Beneficiaries of Social Spending (SISBEN, in Spanish), which is based on assessment of living conditions of individual families. It has been extensively used to target subsidies for health insurance, scholarships, conditional cash transfers, public works, youth training, subsidies for elderly poor, and other subsidies by national and local governments, since 1994. By 2002, 27 million people (60 percent of national population) were registered in SISBEN databases, of whom about 13 million received benefits, at a cost of about US$940 million dollars (1.1 percent of GDP), annually. The purpose of this report is to review the experience of SISBEN to: (a) identify the rationale for introduction of SISBEN (that is, why introduce individual or family proxy-means testing when other, possibly cheaper, geographic targeting system is available); (b) identify the implementation strategy and the advantages and disadvantages of its decentralized implementation, and use; (c) review how different programs have used SISBEN and what has been its targeting and cost-efficiency; and, (d) identify main design and implementation issues, to suggest measures for improvement, and possible lessons for other countries wanting to implement a similar system. 1 Despite some important issues related to SISBEN design and implementation (reviewed in this report), Colombia has experienced a remarkable improvement in targeting indicators in the last few years. The share of subsidies received by the bottom 20 percent of the population increased from 39.3 percent in 1993 to 44.9 percent in 1997 in primary education, and from 29.1 percent to 34.8 percent in health and social assistance, while the share of the bottom 40 percent increased from 65.0 percent to 73.4 percent (primary education) and from 57.3 percent to 65.1 percent (health), in the same period (Table 13). While this can not be attributed exclusively to SISBEN use, it is to be noted that the introduction of SISBEN has been important in determining this outcome. The programs where benefit incidence has been the highest have been those targeted with SISBEN, such as the Subsidized Health Insurance Regime (SHIR), which was benefiting over 11.4 million poor and vulnerable people by end of 2002. The cost of SISBEN design and application has been modest in absolute terms (about US$0.21 per person in the registry, US$0.52 per beneficiary), and relative to the total amount of resources that have been targeted with SISBEN. It has been estimated that to target US$100 dollars to a beneficiary cost less than US$70 cents. For some programs, such as the Conditional Cash Transfer (CCT-Familias en Acción), the cost of SISBEN is about 0.5 percent of the total cost of the program (assuming this is the only program using SISBEN). By comparison, the cost of making payments to beneficiaries in the CCT program is about 3 percent of the cost of the program. The report consists of eight sections and three short annexes. Section 2 reviews the rationale for targeting in Colombia, including legal, institutional and policy mandates of different governments, since the beginning of the 1990s. Section 3 reviews SISBEN's design features including, estimation of SISBEN's welfare index, the questionnaire to gather information, and implementation procedures. Section 4 reviews actual experiences with SISBEN implementation by municipalities, based on a government study made in 2000. Section 5 reviews different SISBEN uses by national and local agencies and programs, including estimates of targeting and cost-efficiency. Section 6 reviews evaluations about the effectiveness and outcomes of SISBEN in the different programs. Section 7 reviews recent changes to SISBEN's welfare index, questionnaire and implementation arrangements, which started to be implemented in 2003, nationwide. Finally, Section 8 presents summary and recommendations of the report. Annex 1 contains a brief review of legal underpinnings of targeting and SISBEN. Annex 2 contains a summary of operation manuals, while Annex 3 contains variables and weights used in the poverty index. 2 2. Rationale for Targeting and Choice of Instrument in Colombia A. Legal and Institutional Aspects The 1991 Constitution decentralized social expenditures (health, education, potable water, social assistance, other) to departments and municipalities (territorial entities), and mandated that about 60 percent of spending be distributed to those entities based on the number of people with unsatisfied basic needs (NBI). Based on this mandate, Law 60 of 1993 defined the distribution formula to be applied, starting in 1994. Article 30 of said law defined targeting as "the process by which social spending is effectively distributed to poorest and vulnerable people", and mandated the government to issue a CONPES (Document approved by the Council of Ministers and President) to design instruments for applying targeting processes at all government levels. Thus, departments and, especially municipalities, which receive a great part of total social transfers on the basis of the number of poor people living in their territories, are mandated to locate and find those people to target them with that spending. A CONPES document defining SISBEN as main instrument for targeting social programs to poor and vulnerable groups was issued in 1994. Law 60 and the national government plan authorized central and territorial governments to provide direct subsidies to beneficiaries to enable them to buy social services, thus, creating a competitive environment for service providers, rather than continue financing historical budget allocations to public suppliers of services, such as hospitals, schools, etc. Further, the health sector reform Law 100 of 1993 and further regulatory Decrees mandated the use of SISBEN to select beneficiaries of the Subsidized Health Insurance Regime (SHIR), created for the poor and most vulnerable without the capacity to pay for health insurance. Following Law 60 guidelines, a new CONPES was issued in 1997 that restated the role of SISBEN as a key targeting instrument and, due to the importance of SISBEN for the SHIR, it recommended a thorough evaluation of SISBEN implementation, the welfare index and application procedures by municipalities. The evaluation was made in 2000, and is the basis for most of the discussions in Sections 4-5, below. Based on the results of the evaluation, a CONPES was issued in 2001, mandating changes to SISBEN welfare index, questionnaire and application procedures to be followed by municipalities and national government. Finally, Law 715 of 2001, which reformed Law 60 of 1993, mandated municipalities to provide funds for SISBEN maintenance and updating. A summary list of legal regulations and foundations of targeting policy and instruments in Colombia is presented in Annex 1. B. Choice of Instrument (Why Proxy-Means Testing) In Colombia, a geographical targeting instrument (known as the Estratificación Socio-Económica, ESE) has been in operation since 1965. This system classifies neighborhoods and rural areas in six strata, 1 to 6 (from poor to rich) based on the external characteristics of houses and neighborhoods. It is applied by municipalities for urban and rural areas following procedures and guidelines provided by the National Planning Department (DNP). It is based on a questionnaire that is applied in the field 3 by trained people and a weighting system (not known publicly) to get to the six strata. This system has been used to target water, electricity and other public services, and infrastructure subsidies. Although there have not been formal evaluations (in targeting efficiency, costs), the system was considered not appropriate to target many of the demand subsidies introduced by Laws 60 (decentralization) and 100 (health reform) of 1993. The use of the ESE system would have been too costly and unaffordable since about two-thirds of people are classified in strata 1-3. The problem is particularly acute in rural areas where most people are classified in strata 1-2. In addition, incidence analysis of spending targeted with ESE shows that spending has been regressive or only marginally progressive indicating flaws of the targeting instrument. Another traditional targeting method has been "means testing" by social workers or other trained personnel, which has been used by institutions, such as the Colombian Institute of Family Welfare (ICBF), for child care, and public hospitals (for fee waiver or reducing care costs), among other institutions. Social workers assess applicants socio-economic conditions on the basis of questions answered by applicants and sometimes on the basis of a home visit. Hospital welfare offices rely mainly on answers responded by applicants and or family members, and other considerations, such as, occupation, verbal skills, personal presentation, place of residence, ethnicity, etc. While these methods may be applied in special circumstances, these are subjective and expensive to apply, and not best suited to support programs which attempt to benefit a large number of beneficiaries, such as the SHIR, or the CCT program. For the above reasons, DNP, through the Social Mission Group, a UNDP supported project created to provide technical assistance to departments and municipalities, and to design targeting instruments, introduced SISBEN in 1994. As will be explained below, SISBEN is a proxy means testing system that classifies people from poor to rich on a score scale. For data gathering activities, and to better capture the poor, SISBEN application combines geographical targeting with family assessment to determine eligibility to a number of benefits provided by all government levels. Simple means testing is not possible given the large size of the informal sector (over 40 percent of labor force), making it impossible to independently verify incomes and assets, and there is a high degree of under-reporting of income (the extent unknown) in the formal sector by both employers and workers to reduce social security contributions. C. Rationale for National-Based or Decentralized-System While in small countries a centralized strategy of data collection and selection of beneficiaries may have several advantages, Colombia is a vast country with over 1,050 municipalities and approximately 44 million people, in which a decentralized strategy could have some benefits. In addition, following the Constitutional mandate for decentralization, it was considered that responsibility for selection of beneficiaries of social programs was best left with municipal authorities. It was postulated that having local authorities those responsibilities would made them more sensitive to social problems and social policy concerns, and more inclined to act on those problems and concerns. This would further strengthen the democratic process started with the election of municipal authorities in 1986. However, major concerns were expressed regarding 4 the possibility of manipulation, favoritism and misuse of SISBEN by local authorities. These concerns were to be addressed with detailed description of implementation procedures, supervision and control activities, and use of uniform data entry and processing software, including automatic calculation of point scores. As seen later in this report, many supervision and control activities have not been properly executed. Thus, the DNP-Mision Social group undertook the task of developing a targeting system with the following characteristics. First, it needs to provide uniform, objective and transparent criteria for all municipalities to ensure that every Colombian is treated (classified) equally no matter where his (her) place of residence is. Having Colombia a unitary government, and being fiscally centralized (that is, most taxes are collected by central government), it is a government objective to equalize spending treatment across the country. Second, the system has to be transparent to all (municipal authorities, communities, beneficiaries), so that everybody knows who has been selected and the rules for that selection process. Third, procedures have to be reviewed periodically to correct problems and possible misuse of system. The following section provides a detailed presentation of SISBEN. 5 3. The SISBEN: Design and Implementation Features A. What is SISBEN SISBEN is a general purpose system for selecting beneficiaries for social programs in Colombia. It has a statistically derived proxy means test index that serves as an indicator of households´ economic well-being. The variables that determine welfare include availability and quality of housing and basic public services, possession of durable goods, human capital endowments and current income (this latter variable was excluded in the new revised SISBEN Index due to unreliability and lack of predictive power, as seen in Section 7). The system includes a set of norms and procedures defined at central level and operated at municipal level to gather information necessary to calculate the welfare index and select beneficiaries for the numerous social programs. B. Objectives of SISBEN The general objective of SISBEN was to establish a technical, objective, equitable and uniform mechanism for selecting beneficiaries of social spending to be used by all government levels. The specific objectives include1: i) classify applicants to social programs in a rapid, uniform and equitable way; ii) strengthen institutional development of municipalities with the establishment of a modern social information system; iii) support inter-institutional coordination within the municipality to improve impact of social spending, avoid duplicities and concentrate efforts on the poorest; and, iv) elaborate socioeconomic diagnostics of poor population to better prepare social development plans and projects for poor people, and facilitate attainment of targeting goals for departments and municipalities.2 C. Institutional roles and responsibilities The institutional roles and responsibilities are defined in Table 1. As will be presented later in this report, from Table 1 some important deficiencies in the SISBEN system include: (a) lack of central level monitoring of application procedures (including monitoring the use of cartographical information and poverty maps for selection of poor areas to be surveyed); (b) lack of auditing of databases; (c) infrequent evaluations; and (d) lack of consolidation into a central database. Some of these deficiencies are being addressed in the new SISBEN application, including auditing of databases and consolidation of municipal databases into a consolidated national database, by DNP. Auditing of application procedures is planned to be done with the participation of departments, but it remains to be seen if these entities have the capacity to do so effectively. In addition, there are no plans for further evaluation of SISBEN in the near future. 1Manual No. 1 of SISBEN (DNP-UDS-Misión Social), 1994. 2It is important to note that the last (1993) National Population Census was contested by many municipalities and that SISBEN Census was a source of information that many municipalities used to help design their government plans. 6 Table 1. Roles and Responsibilities of Different Government Levels for Design and Application of SISBEN Activity National Department Municipal Government Government Government Setting of targeting policy (Provides X X (for Department X (for Municipal criteria for national and territorial Programs under Programs under programs) national norms) national norms) Design of SISBEN · Determine variables for score X -- -- · Design questionnaire X -- -- · Prepare operation manuals X -- -- · Develop data entry and processing X -- -- software Implementation of SISBEN · Provide training for application X (Dpt. X (local system -- administrators) administrators) · Provide Technical Assistance (TA) DNP to Dpt. and Dpt. To -- municipalities. municipalities. (In practice not done). · Provide financing for application DNP, MOH, -- Municipal budgets municipalities · Monitoring application procedures DNP (in practice -- -- not done) · Auditing of data bases DNP (in practice -- -- not done) · Consolidation of Central Data Base DNP (new SISBEN) -- -- · Periodic Evaluation of SISBEN DNP (first done in -- -- 2000) Source. DNP, Misión Social. SISBEN Administration Manual. 1994 D. The Components of SISBEN SISBEN has two main components. The first is the welfare index or SISBEN Index which determines the welfare level of the family on the basis of a statistical model using a small number of variables. The second are the procedures to gather information on those variables, to guarantee its quality, entry data into computers, and for management of databases. The following is a detailed description of each of these components. (1) Calculation of the SISBEN Index and the selection of variables and weights For calculation of the SISBEN Index the following procedure was followed. Using a socio-economic representative survey applied to about 25,000 families nationwide, statistical models were fitted to identify the variables that better predict welfare of the population. Thirteen variables were identified as providing the better information for calculation of the SISBEN Index. The SISBEN Index gives a continuous score from 0 to 100 (from poorest to richest) divided into six brackets or levels (Level 1 to level 6) to facilitate application by territorial entities. Levels 1 and 2 are people in poverty, and are the subject of most national and local programs. The same procedure was followed to update the SISBEN Index with the 1997 Survey of Living Conditions (ENCV, in Spanish) applied to about 10,000 families for the new SISBEN which started to be implemented in 2003, countrywide. 7 The SISBEN Index is estimated using both qualitative (categorical) and quantitative variables. For this, the statistical algorithm of Qualitative Principal Components (PRINQUAL), which assigns numerical values to categorical variables, was used to make it possible the combination of qualitative and quantitative variables in principal component analysis. The quantification is made in a way that maximizes the variance of the first principal component of the whole set of variables. Once the categorical variables have been converted to quantitative variables, the principal component analysis is applied.3 The PRINQUAL procedure allows the identification of the most important variables to define the standard of living. The criteria for selecting each variable are the discriminatory capacity of each variable against the standard of living and its capacity to explain the variance. The group of final variables selected by the algorithm was the one with more discriminatory power about household welfare differences.4 The estimated index is composed of four factors, as follows: 1) housing quality and possession of durables; 2) public utility services, 3) human capital (education) levels, and 4) family demographics, unemployment, dependency ratio and income per capita. Within each factor there are variables that are weighted to assign one unique score by household. The weights are different for urban and rural areas. The cut-off points to determine poverty Levels 1 and 2 were determined using probit estimations to maximize the likelihood of correctly classifying poor families, having as comparators or reference lines the NBI and income Poverty Line (LP).5 The cut-off for level 1 corresponds to extreme poverty (two NBI and per-capita income below the cost of food basket), while level 2 cut-off corresponds to "poverty" (one NBI and income between one and 1.7 times the cost of food basket). Table 3A in Annex 3 presents the detailed results of the statistical estimates. The new SISBEN Index cut-off points were calculated following the same statistical method. The Questionnaire The questionnaire is a two-page format that includes the variables selected for the SISBEN Index, and some additional variables to identify and characterize the family. The SISBEN questionnaire has 62 questions (74 the new one) organized in 7 sections. The sections are described in Table 2. The revised SISBEN Index and questionnaire are described in Section 7. Identification numbers used In Colombia, there is not a social security number, and while most Colombians have the citizen identification card (ID), many (specially poor, indigenous people and other ethnic groups, and children) do not have this document. In the absence of a unique ID number for all people, SISBEN questionnaire (Ficha de Clasificación 3 Other method for "quantification" of categorical variables is to use expert's judgments to order the different categories from, say, 0 to 100. This method is often used, but has the problem that experts generally use linear orderings of the different categories when non-linearities may be really present. 4Similar to other estimation methods (regression analysis, for instance) there are predictions errors which are higher when predicting the welfare of individual households. Thus, predictions to the household level have to be taken with precaution, as they have large confidence intervals and are, thus, unprecise. 5Castaño (1995). 8 Socioeconómica) has a four-digit identification number which is used to identify households and families within municipalities. This number is given by the municipality and is used for verifications and checking of information. Within the household, every member is identified with order number, relationship with head of household, own ID number (if available). Children are identified with the number of civil registry (if registered). These household and members ID numbers, plus village, municipal and departmental codes, provide a unique number which allows for verification of duplicates at national level, if required. However, as seen later in this report, severe problems have occurred with duplicate information, resulting not much from the lack of ID numbers, but mainly because when questionnaires are updated a new questionnaire number is provided by the municipality. Also, the application software does not allow updating of information under the same File and does not keep a record of updated files. These problems are expected to be corrected in the revised version of SISBEN and software. Table 2. Variables Included in SISBEN Questionnaire* Face Section Name of the Section Questions A 1 Housing identification 9 questions (4-12) 2 Information about housing and services 9 questions (13-21) 3 Information about nuclear families: 11 questions (22-32) 4 Control of work** 7 questions (33-39) B 5 Socio-demographic antecedents 13 questions (40-52) 6 Education: 4 questions (53-56) 7 Occupation - income*** 6 questions (57-62) * Refer to the questionnaire in use until December 2002. The new questionnaire is similar but includes more details for identification of areas, villages, etc. and a few others described in Section 7, below. ** Means questions that assure quality of field work and describes whether survey was on-demand, by outreach method, who did the survey, etc. *** Income variables continue in the new SISBEN application but they are no longer weighted for Index. Source: DNP, Misión Social. SISBEN Interviewer's Manual, 1994. Definition of household unit SISBEN distinguishes households from families. While the household is the traditional survey unit in Colombia (used by the National Statistical Department, DANE for all censuses and sample surveys), the family unit is used by many social insurance and assistance programs, such as the SHIR. SISBEN defines household as a person o group of persons that live in a house or part of it and share food or food budget. The family, akin to nuclear family, is the person or group of persons within the household that live permanently in the house, and includes the couple and single sons and daughters with or without income, and people with no dependants and no income that depend from family head. This definition is formally stated in the interviewer's manual, and interviewers receive training for making the appropriate distinction. The concept of family has been, however, difficult to apply by interviewers and will be eliminated (leaving only the household unit concept) in the new application of SISBEN. 9 Household members The questionnaire includes basic information of each member of household such as, names, last names, gender, civil status, family relationship, date of birth, ID number, affiliation to social security, assistance to formal and regular education center (all members), type of education center (public, private), last level of education approved. There are also questions on occupation and income of all family members. Occupation refers to main activity during last two weeks prior to survey and normal occupation during last year. Also included is type of job and size of firm. Incomes include monetary income received from work, grant or transfer from others outside the household, in the last month. When income varies every month, the amount to be registered is the average income per month. Income is asked for all family members (all ages, including children because they may also get transfers). There are changes in some of the variables of the questionnaire in the revised version of SISBEN, as explained in Section 7. Assets The questionnaire requests information on ownership of assets, such as ownership of the site ("sitio') in which the housing unit is built, and of durable goods, including refrigerator, TV, fan, washing machine, and blender.6 The new SISBEN questionnaire includes additionally: cable TV, water heater, oven, air conditioning and excludes blender. Characteristics of house The questionnaire inquires for the quality of housing through the presence of substandard materials for roofs, walls, floor, and access to basic services, such as potable water, electricity and cooking fuel or materials. (2) Strategy and procedures to collect the information For the Initial construction of the SISBEN data base of potential beneficiaries and massive updates Municipalities implement SISBEN following two steps, as indicated in operation manuals produced by DNP (see list of manuals and summary content in Annex 2). In the first step, municipalities identify poor areas to be surveyed, both rural and urban, using a variety of information to produce local poverty maps. In the second step, municipalities launch the survey operation (with own staff and resources, or with contractors) to apply SISBEN questionnaires to ALL residents in selected areas. People not surveyed because they live in non-selected poor areas can apply for SISBEN application at SISBEN municipal office. By legal mandate, municipal offices are obliged to register those applicants. The new application of SISBEN follows these described two steps, and the initial outreach or "barrido" is expected to cover over 90 percent of people to be registered. This method was preferred by DNP (for the 1994 6Ownership of the site is important because poor families may own the "housing unit", but not the site which can be the subject of irregular settlements or invasions. 10 and 2003 applications), although there are advantages and disadvantages of outreach relative to registering only those applying for benefits, as seen in what follows: The Outreach Method- Survey of Poor Areas The Application Method-Survey of Those Applying for a Benefit Advantages Advantages · There are higher chances of getting the · Total number of households to be interviewed poorest of poor who typically are ignorant determined by number of program about programs and or have no money for beneficiaries, and, thus, total registry cost may transport, other, to apply for programs; be lower, although per unit household costs · Can provide a great number of prospective are higher; beneficiaries very quickly to start a large · A smaller data base that is easier (cheaper) to program; manage and update. · May be more transparent to people and the public in selection of beneficiaries. Disadvantages Disadvantages · Total cost of registry many be higher, · Covers those applying for benefits, but these although per unit cost cheaper since a census- may not be the poorest---may live closer to like operation covers neighboring households urban areas, be more informed, have higher saving in transport costs, other; education and or have money for transport · Large data base difficult to manage and keep costs, etc. updated. · Per unit costs are higher (interviewers need to travel long distances for home visits), there are problems with locating addresses of poor. Best suited when: Best suited when: · Poverty levels are high (over 50%), poverty · Poverty levels low (say, below 20%), poverty areas are homogeneous (rural, outskirts of areas are heterogeneous; cities), low education of people. · People have high education levels and · There is a need to start a large program outreach campaigns to encourage program quickly and government needs to be pro- participation can be implemented. active in reaching poorest. There are detailed instructions on how municipalities should organize cartographic information for selecting areas to be surveyed using information from DANE and other municipal information. However, as seen later in this report, there is no information on the extent to which municipalities follow those procedures to ensure proper coverage of poor areas.7 Following are administrative aspects that municipalities have to comply with for the proper application and administration of SISBEN. In order to implement SISBEN, municipalities have to: · Create a Technical Committee (TC) to administer SISBEN. The TC is composed of members of social and planning secretaries and a delegate from the mayor who presides it. The TC coordinates all logistical aspects and budgetary matters. 7Ideally, DNP should have provided each municipality with a detailed poverty map prepared following a common methodology to reduce the possibility of manipulation of survey areas, and ensure that municipalities choose the poorest areas for their survey work. One way to do this is by combining the estimates of the proxy means test model with census information to produce disaggregated (village level or lower) municipal poverty maps (Hentschel, J., J. Lanjouw, P. Lanjouw and J. Poggi, 2000). This, however, was ruled out as the latest census in Colombia dates back to 1993, and the results were highly contested by municipalities. 11 · Provide office space and necessary inputs for the operation of SISBEN. The size of office space and amount of inputs (including staff) depends on number of people to be registered. · Appoint SISBEN administrator responsible for planning and executing necessary activities for training, data collection, data entry and building data bases and keeping then updated. He (she) also is responsible for distributing data base to user agencies and for solving complaints. Specific responsibilities of SISBEN administrators and staff are the following: · Identify poor areas to be surveyed, make estimates of approximate number of people to be covered and design and apply strategy for application of survey. Common sources of information to determine poor areas are: a) the socioeconomic stratification (ESE) made by the municipality, to provide subsidies for public services, b) information about NBI provided by the National Statistics Department (DANE) based on the latest census (1993), c) information about marginal areas and risk groups provided by social secretaries, and d) cadastral records for property tax, and others, managed by municipalities. · Plan for logistics to apply the survey. Includes: a) preparation of budget for matters such as, printing and or copying of questionnaires and manuals, transport, staff required for gathering information, supervision, and data inputting. If survey is to be contracted, preparation of terms of reference for contracting, evaluation of proposals, signing of contracts (done by mayor) and supervision and auditing of contracts. The administrator also must ensure availability of proper computational equipment, and availability of physical space with inputs required. Data collection Data collection is done in much the same way as a census or a household survey is done in Colombia. The process includes: · Based on detailed cartographic maps, interviewers are assigned specific routes with sectors and addresses of houses to visit. · Properly identified interviewers request information from head of household, if at home, or housewife or person older than 18 years, who are qualified informants. When questionnaire is finalized, head of household or other respondent signs it. The respondent is advised this is a public document that carries legal penalties for false information. Supporting documents requested are ID cards, birth certificates and, sometimes, water or electricity bills to certify socioeconomic strata. No verification is made of employment, occupation status and income. · Once questionnaires are filled, they are revised daily by supervisors who seek to detect mistakes, unfilled entries, coding errors, and inconsistencies of information, using a validation matrix. Questionnaires with complete 12 information are entered into database. Questionnaires with wrong or missing information are re- checked in the field. · The data entry and processing software (distributed along with operating manuals by DNP) has validation matrixes to further detect errors of range or inconsistency of variables to be corrected in the field before information is entered into the database. The software also detects duplicities by ID number and gives a list to be corrected. Questionnaires passed on to the historic file are automatically assigned the poverty score pertaining to household and or family, that can be used for beneficiary selection for the programs that use it. Those excluded from the initial "barrido". People living in non covered areas can apply for registration in SISBEN. They can go to SISBEN offices where they are registered with names and address to be surveyed, according to a municipal plan. The municipal SISBEN office is a permanent office in most municipalities and people usually go there looking for application or re- application of survey. SISBEN offices distribute ID cards of SISBEN with name, ID number of person and family group and the score. The same score applies to all family members. Who conducts the interviews Interviews are conducted by trained people, usually high school graduates. These are not usually social workers or specialized professionals or city employees. They are usually temporary workers paid on the basis of the number of house visits per- day. When survey is conducted by contractor firms, they are charged with contracting interviewers and training them, according to SISBEN manuals. What is their training Initially, when SISBEN was introduced, there was massive training by DNP- Mision Social to Municipal SISBEN administrators, department employees and even private individuals and firms. A one-week long course was designed to cover all 5 manuals (7 in new version) for administration and planning of SISBEN operation, taking up surveys, supervision and quality control, and data entry and data management activities. Some courses were given in association with universities, and at the end of the course, people were certified to take SISBEN surveys in municipalities. Over 1,000 people are estimated to have been trained in the initial year of implementation. After the initial massive survey, SISBEN administrators or contractors are charged with training people for updates and taking surveys, although the extent to which this has been done and the quality of training, varies greatly by municipality. Cost per interview As part of the administration guidelines for SISBEN application and operation, DNP provides benchmarks for number of interviews per day for urban and rural areas, as well as average costs per interview. An interviewer in urban areas can make about 20 interviews per day (8 hours of work), spending on average 25 minutes per questionnaire. The cost per interview was estimated at US$2.25 for 1995, while that for 13 2002 at US$2.7.8 Costs per interview for updates or on-demand survey are over 60 percent higher than the above costs for both areas. This occurs because interviewers have to cover different places or neighborhoods during the day increasing transport costs and reducing the number of surveys they can do in a day. Costs per interview vary greatly by municipality depending on the number of families to be surveyed, the capacity of the municipality to negotiate with contractors, the transparency of the contracting process, among other factors. Computer system and needs A critically important decision by DNP-Mision Social when SISBEN was first launched was the provision of data entry and processing software. The application software includes over 30 routines to check for range and consistency of variables, and calculates point scores automatically. According to procedures, data entry has to be done every day, so that if inconsistencies and errors are detected, interviewers can correct them immediately in the field. The application software is easy to administer and use, to install in low-capacity computers, and was distributed to municipalities (and contractors) free of charge, along with the user's manual. The purpose was that computational requirements were not a barrier for municipalities to implement the program. Who manages database The SISBEN database is managed and administered by SISBEN administrators in each municipality. He (she) is charged with building the initial data base, updating it and providing information to municipal authorities (other secretaries such as health, social welfare, education, etc.), and to national authorities and programs. Links to other databases There is not a centralized data base, although one will be constructed in the new application of SISBEN. National programs such as the CCT-Familias, built own data base by merging (and cleaning up) data bases of municipalities included in the program (about 625 out of 1,050) as described in Section 5. The SISBEN data base is not formally linked with other data bases. The original design (and software) was for the municipality to have its own data base and use it for local programs, or national programs which involve selection of beneficiaries by the local level. The revised SISBEN and software provides for DNP building a central data base by merging municipal data bases. Who has access to SISBEN databases SISBEN data bases are the property of municipalities. SISBEN administrators often share data bases with municipal secretaries of health, social welfare, education, and others, depending on the municipality. The means of transfer vary by municipality and include diskettes and or CDs. Also, municipalities that are included in national programs share databases with administrators of those programs who, based on SISBEN scores, select beneficiaries for their programs. In many municipalities, SISBEN database or part of it is installed in public hospitals to verify eligibility for hospital fee 8CONPES 055, 2001. 14 subsidies, and to identify people affiliated to the SHIR and the insurance company (ARS) responsible for payment. Law 100 provides subsidies of 95 percent of the cost of hospital care for non-affiliated Level 1 patients, 90 percent for Level 2 patients and 70 percent for Level 3 patients. Possibility and procedure for appeals People not included in the initial survey have the right to be interviewed and included in the data base if they ask to be included. This is well known in Colombia and people take the opportunity to exercise this right. This does not mean that people will be included in social programs or that they have the score they expect. As presented in Section 5 of this report, there are a great number of appeals either to be re- interviewed or to change score points. Although the number of cases relative to the number of households in the data base varies greatly by municipality, most people that appeal use legal means to press municipalities and SISBEN administrators to respond quickly. The first is the constitutional provision known as the "Petition Right" whereby any Colombian can ask to be interviewed or re-interviewed to have a SISBEN score. The mayor then has ten working days to answer in writing indicating the course of action he (she) is going to take (interview, re-interview or deny the petition). The second is the "Tutela Right" by which Colombians can go to a judge for protection of a basic constitutional right. In SISBEN's case, the right most often invoked to be protected is the "right to life" (Derecho a la Vida); the reason being that SISBEN determines who is and who is not eligible to be affiliated to the Subsidized Health Insurance Regime (SHIR). This Tutela Right is often invoked by people who are sick of AIDS, cancer and other catastrophic health problems which are insured in the SHIR. These two legal provisions are amply known and used by the population, including the poor, and the cost of legal action is usually low. Most municipalities, especially the larger ones, have one or more legal counsels or lawyers to answer legal requirements. Updating of Household Information The 1994 CONPES introducing SISBEN and administrative manuals indicated the need for updating of information every three years. Although it was not clearly stated what was the information that needed to be updated, it was understood that every three years people should be re-interviewed and reclassified in their poverty levels. The rationale for the three-year period (although not based on empirical evidence) was that living conditions of people would change in this period of time, especially for people receiving social programs. As indicated in Section 4 of this report, most municipal data bases have not been updated, although it is impossible to determine the extent of this phenomenon. Anecdotal evidence suggests that updating has been especially problematic in big cities, such as Bogotá. In smaller municipalities, updating appears to have occurred mote often with change of local administrations. One major problem with SISBEN Operation Manuals was they did not specify which information needed home visits for the update and which did not, for changes occurred to the household within the three-year period. This led to the decision by municipalities to file a new questionnaire (and file numbers) for every change, from the birth of a baby or death of a family member to a change of address. While the change in 15 address would surely require the family to be re-interviewed (since points scores are heavily influenced by housing conditions), the birth of a baby or death of a family member could have been handled in the SISBEN office (a permanent municipal office), upon presentation of birth or death certificate, thus, avoiding a large number of unnecessary and expensive home visits. Further, when the family was re-interviewed it got a new SISBEN number (with no connection to the old number) making it very hard to manage historical information of the family and eliminate duplications. Many of these problems are expected to be corrected in the new SISBEN application. (3) Verification and audits There are several supervision and auditing rules for implementation of SISBEN, which are described in the Data Quality Control Manual. The supervisor is the person charged with reviewing filled questionnaire information. She (he) detects and corrects mistakes and cleans off inconsistent information. At the end of every working day, the supervisor reviews 100 percent of questionnaires, sending those without any mistakes to data entry, and those to be corrected to supervision for reapplication. A supervision report with most common mistakes is sent daily to the administrator of SISBEN, so that he (she) can improve interviewers´ training. Further, the SISBEN application software produces a list of errors of consistency and range, as well as a list of duplicate questionnaires. All questionnaires with errors must be verified in field (home visit) unless the error is of data entry. In addition to previous supervision activities, SISBEN administrators randomly select 20 percent of surveys for verification of information collected. The survey is re- taken and if information does not coincide with the initial collection, the new information is adopted and inputted into the computer. A report is filled against the original interviewer to detect failures of training and or mis-behavior and take corrective actions. However, it is not known to what extent these procedures are followed by municipalities and or contractors. 16 4. Implementation of SISBEN A. People registered in SISBEN Data Bases By 2000, the latest year for which consolidated information was available for this report, there were approximately 27 million people registered in SISBEN municipal data bases, nationwide. This represents about 63 percent of total population of the country, 57 percent of the urban and 80 percent of the rural population, respectively. The year when more people were registered was 1995 when the SHIR started to operating massively, and when the Ministry of Health (MOH) provided funds for a speedier application of SISBEN. According to figures from a recent evaluation of SISBEN study, over 95 percent of the 1,050 municipalities implemented SISBEN for the first time in 1994-95.9 This latter figure was much lower (about 80 percent) for the smaller municipalities (less than 20,000 inhabitants). Of the total number registered, about 65 percent are classified in Levels 1 and 2 and about 24 percent in Level 3, while the number registered in 4-6 levels is only about 10 percent. Tables 3 and 4 show the evolution of application of SISBEN in both urban and rural areas since 1993, and the distribution by SISBEN levels 1 to 6. Table 3. People Registered in SISBEN by Urban and Rural Area, 1993-2000 Year Urban % of Rural % Rural Total Urban Population Population 1993 and 1.973.748 8 1.168.905 10 3.142.652 before 1994 1.118.493 4 13.971 0.1 1.132.464 1995 4.120.831 15 2.637.013 23 6.757.844 1996 1.265.259 5 533.853 5 1.799.112 1997 2.253.744 8 579.199 5 2.832.942 1998 2.542.772 9 2.111.599 17 4.654.371 1999 2.585.595 9 1.732.267 14 4.317.863 2000 1.152.729 4 966.308 8 2.119.037 Total 17.013.171 57/a 9.743.116 80/a 26.756.286 /aCorresponds to the percentage of accumulated people registered over estimated population for 2000. Source: DNP et al. (2001). Table 4. Distribution of SISBEN Population by SISBEN Level, 1993-2000 Level People Registered % Level 1 6.592.247 24.6 Level 2 10.841.296 40.5 Level 3 6.469.761 24.2 Level 4 2.223.855 8.3 Level 5 613.812 2.3 Level 6 15.905 0.1 Total SISBEN 26.756.286 100.0 Source: DNP et al. (2001). 9 The Ministry of Health, through Decree No. 2357 of 1995, authorized the use of one percent of the funds of the Solidarity Sub-Account of the Social Solidarity Fund (FOSYGA) to give financial and technical assistance to municipalities for implementation of SISBEN. 17 The following issues are to be noted from these Tables: · There are problems with quality of the database, especially at the beginning of the survey process. There are a great number of people who appear with interview dates earlier than 1993 when the system was first launched. This may reflect difficulties in using the application software provided. Reportedly, many municipalities had problems merging different data bases. These problems are expected to be corrected in the new application software provided for the new application of SISBEN. · There may be (and most likely are) a lot of duplicate families in the registry within and among municipalities. As indicated previously, when a family is re-interviewed for updating information or for responding to judicial mandates, families are given a new number and there is no easy way to have a historic file of each family and or to produce non-duplicate family files. · Application of surveys has varied greatly every year responding, to a large extent, to irregular financing and, to a lesser extent, to local political cycle. Anecdotal evidence suggests that mayors tend to do surveys at the beginning of their administration, dismissing the work done by previous administration, and hoping to include new beneficiaries. · There is a great number of people (6.5 million people or 24 percent of total registered in SISBEN) who are in Level 3 and, thus, can be categorized as "near-poor", or people who have been estimated a score higher (in many cases only marginally) than Level 2 score. This is the group potentially unhappy with SISBEN scoring method and that is expected to complain and ask for re-interviews at the SISBEN local offices. In this group, we expect to find the majority of the errors of exclusion. B. Administrative Aspects of Implementation A study conducted by the MOH in 2000 to assess the implementation of SISBEN on a sample of 100 municipalities (50 percent department capitals, typically over 100,000 inhabitants, 25 percent medium size, 20,000 to 100,000 inhabitants, and 25 percent small size, less than 20,000 inhabitants), found the following: 10 · Technical committee for coordination of SISBEN activities. Found in only 58 percent of municipalities. · Administrator of SISBEN. Appointed in 94 percent of municipalities; 52 percent were permanent staff, 30 percent were hired on a fixed-term contract and 13 percent were temporaries. About 62 percent of administrators had been in that position for more than 1 year. About 37 percent of administrators had higher education level, 29 percent had not finished higher education, and 34 percent had only high school. All these indicators are poorer in small municipalities. 10Ministerio de Salud (2000). 18 · Training. Initial training provided by DNP-Mision Social was reported to have been critical for implementation of SISBEN. More than half of municipalities indicated to have received training. Later (after initial) training was provided by DNP and departments when demanded by municipalities. · Office space, equipment and inputs. About 73 percent of municipalities had adequate space for SISBEN office, 93 percent computational equipment, 85 percent manuals, 42 percent operating budget, and 48 geographical information maps. This latter finding is particularly worrisome since, in the absence of good cartographical information and a good poverty map it is doubtful that municipalities can carry on an adequate and objective determination of areas to be surveyed and, thus, there is ample room for manipulation by local authorities. The problem is aggravated because there is no monitoring of how areas are determined by municipalities from the departmental or central (DNP) SISBEN office. The new SISBEN application does not address this issue. · Who applies survey. Most municipalities (specially the smaller ones) prefer to do initial survey and updates with staff of their own. 75 percent use own staff, 12 percent use contractors or hire temporaries, and 3 percent use both alternatives. Larger municipalities (over 200,000 people) use contractors, generally. C. Monitoring and Evaluation of SISBEN Implementation Monitoring and evaluation activities of SISBEN implementation in municipalities have been very poor. Although this responsibility was assumed to rest with DNP-Misión Social (directly or by contracting it out with private sector or universities), it was not established clearly by the CONPES document or the operating Manuals, and has not been done. As a result, a study of Ministry of Health (MOH) on 3.213 randomly selected filled questionnaires (Fichas) that were re-applied in home visits (with strict verification of answers) found some worrisome signs of possible misapplication and or manipulation. These were: · About 4 percent of the sample surveys misclassified rural and urban areas. This is important because weights and scores are different between urban and rural areas, and people could use misclassification to manipulate score values. · Only about 48 percent obtained the same score or Level of SISBEN, 44 percent obtained a higher level than in the previous survey, and 8 percent obtained a lower level. Although the re-interviews were done a few years later (when living conditions are expected to have improved) this may be an indication of some manipulation of beneficiaries to obtain lower scores and be beneficiaries of the SHIR. In addition, even if the differences were due to improvements in living conditions (rather than manipulation), this still signifies a need for more regular updating. 19 · Although almost all of municipalities in the study responded having filled paper questionnaires (to support the information inputted into the data base), only 62 percent of filled questionnaires were, in fact, really kept. Reasons for not keeping questionnaires included damage, loss or store failures in warehouse, among others. Procedures for file keeping and the minimum time to maintain files were, however, not clearly specified in operation manuals. Most of these problems are expected to be corrected in the new SISBEN application. The new application software includes a quality control module that allows entering re-surveyed data of a randomly selected sample of 10 percent of questionnaires filled every day, comparing this data with the original questionnaires´ and producing automated reports on the quality of data gathering and entry. The computer program provides a list of errors to determine the reliability of gathered and entered information. The maximum percentage of errors permitted by DNP is less than 10 percent. If it is equal or more than 10 percent, the information is classified as being not reliable. Municipalities will be audited periodically during the implementation process to verify the quality of information using this automated process. It remains to be seen, however, if this process can realistically be implemented in the 1,050 municipalities, and if DNP has the capacity to conduct or oversee it. D. Poor Updating Updating of SISBEN in these nine years of operation has been generally poor. According to the SISBEN evaluation study, although about 79 percent of the municipalities had an updating plan at the time of the study in 2000, only a few (about 12 percent) had done the whole survey again, which was supposed to be done every three years. Most have updated SISBEN records on demand basis--that is, when people ask to be interviewed for the first or subsequent times--to update information or for change of address or other reasons, including judicial mandates. Reasons for this poor updating record include lack of financing and of directives from DNP. About 55 percent of municipalities have updated SISBEN records on demand, 12 percent by surveying poor areas again, while 6 percent by re-interviewing families randomly. E. Citizen Oversight and Social Control While community participation was one key aspect in supervision and social control activities of SISBEN implementation, in practice, this has been limited. The study of MOH found that 76 percent of municipalities had citizen's oversight committees (veedurías ciudadanas), but only about 59 percent of them participated in the identification of beneficiaries' process. Other social control groups that exist in municipalities include, a) committees of community participation (existing in 61 percent of municipalities), and; b) users´ associations of social security system (existing in 48 percent of the municipalities). However, participation of these groups in oversight activities has also been limited. 20 In addition, the MOH study found that only in few municipalities there were formal written complaints. Most complaints were informal for reasons, including, a) fears of being threatened by people they denounce, especially in violent regions of the country; b) lack of time to participate and or complain; c) the presence of corruption or suspected corruption or manipulation, albeit in a few cases, of members of citizen's oversight groups who reportedly asked for money to influence score levels; d) lack of information by the community about institutions and or places to go to denounce irregular or corrupt behavior. F. Municipal Manipulation SISBEN procedures give municipalities complete powers over the application of SISBEN. Mayors decide over the areas that will be interviewed on the massive surveys or "barridos", and over the measures that will be taken to include people on a demand basis. In addition, being so close to the electorate and not being responsible for financing social programs with local taxes (there is fiscal centralization and equalizing poverty-based transfers), there is a local incentive to exaggerate the number of poor to get a greater amount of resources from central government. As a result, there are great risks of manipulation by mayors and the strict procedures and guidelines produced by DNP propose to reduce the scope for the materialization of those risks. Abundant anecdotal evidence suggests that manipulation and or misuse of SISBEN may have in fact occurred, but unfortunately, there is no statistical evidence to quantify it and to determine the possible impact of such behavior. The new application procedures for SISBEN and recent legal measures include penalties for public officials who manipulate information. G. Manipulation by Beneficiaries Also abundant anecdotal evidence suggests possible manipulation by beneficiaries. According to CONPES document that mandated revisions of SISBEN in 2001, people, in some cases, have asked for a home visit and have provided the address of a house they have rented for that purpose. This house meets all requisites for getting low score levels. In some other cases, people may have abused the Petition and Tutela Rights provided by the Constitution to force municipalities to apply interviews or re- interviews, when people do not agree with the scores. While there may be cases where people may have been misclassified, or not registered in SISBEN, there are many cases where people have tried to influence SISBEN administrators to attain low scores. In the revised SISBEN, there are penalties for manipulation or misreporting of information. H. No Central Data Base The original SISBEN design aimed at a decentralized conformation and management of database by municipalities. The aim was not the conformation of a central data base of potential beneficiaries. Legally, municipalities had no obligation to share databases with their department or national governments until 2001, when Law 60 was reformed by Law 715. Now municipalities have to share data bases with DNP. Additionally, the application software was developed to manage small databases and 21 presented problems with large and aggregated databases. Further, the lack of control and supervision, and of a unique identifying number, gave space for duplications, although the extent of duplications in the MOH study was not large (only about 3 percent). In the new application of SISBEN there will be a central audited data base produced by DNP to be shared with national and local agencies. 22 5. Uses of SISBEN for Program Eligibility As indicated previously in this report, SISBEN was designed to serve as a targeting tool of social programs operated by national level, departments and municipalities. The following is a short description of programs using SISBEN by government level, and of how those programs have used SISBEN. Figures on the number of beneficiaries and budgets spent in 2002-2003 are shown in Table 5. In total, programs using SISBEN cover about 12.9 million people (about 74 percent of population in levels 1 and 2), at a cost of about US$941 million dollars (about 1.1 percent of GDP), annually. These figures do not include housing programs, and many other small subsidy programs developed by municipalities that use SISBEN alone or in combination with other targeting systems. In addition, coverage rates are most probably under-estimated as there may be duplications in the consolidated (27 million people) data base.11 Table 5. Main Programs Using SISBEN for Selecting Beneficiaries, 2002-2003 Program Agency No. beneficiaries Amount Coverage of (Thousands) US$ Million SISBEN 1-2 (%) Subsidized health MSP-Municipal/a 11,400 764 55.7 /e CCT-Familias MPS- ICBF/b 1,300 83 8.0 Youth training MPS-SENA/c 35 35 N.A. Public Works Presidency-MPS 67 32 N.A. Elderly poor Municipalities/d 80 27 N.A. Total 12,862 941 0.74 /a MSP means Ministry for Social Protection created in 2003 by merging the former Ministries of Health and Labor and Social Security. N.A. Means: not available. /b Starting in 2003 program operated by National Institute of Family Welfare (ICBF) depending from MSP. Includes 315,000 mothers with 800,000 children. /c Starting in 2003 program operated by National Training Institute (SENA) under the MSP. /d Includes estimates of subsidies to elderly provided by municipalities and co-financed by the Red. /e According to MSP figures, about 15 percent of affiliated are not in SISBEN databases. Total SISBEN 1-2: 17.4 million people. Source: Author's calculations based on figures from the different programs. A. National Programs Subsidized Health Insurance Regime (SHIR) implemented by municipalities This program is the largest user of SISBEN, nationwide. By April 2003, 11.4 million people (about 55 percent of SISBEN 1-2 population) were included in the program. Municipalities select the beneficiaries to be selected in the program from SISBEN level 1 or 2 applicants.12 Annual cost of the program amounted to 764 million dollars in 2002 (0.9 percent of GDP), the single largest social insurance program in Colombia. The program was created in the health sector reform Law 100 of 1993. It is funded with transfers to municipalities and departments, and with a National Solidarity 11In fact, there may be duplicates in both the program beneficiaries (numerator) and the total consolidated SISBEN database (the denominator). We assume, however, that duplicates are larger in the denominator than in the numerator (since programs do clean up their data bases) and, thus, suggest than overall coverage figures of SISBEN levels 1 and 2 (poor) people presented in Table 5 could be under-estimated. 12Most municipalities, especially the larger ones, do extensive information campaigns using local radio, TV, other means, to inform people that have levels 1 and 2 of SISBEN to go to municipal health secretaries to register to the program, and or to renew their annual affiliation. 23 Fund (FOSYGA). It is operated by municipalities, who in addition to selecting beneficiaries for the program, contract and pay private (profit and non-profit) health insurance companies (known as Administradoras del Régimen Subsidiado or ARS, in Spanish), for the provision of a determined health package.13 The health package (Plan Obligatorio de Salud Subsidiado, POSS) covers primary and basic hospital care, plus insurance for selected catastrophic illness for the family. Table 6 presents the evolution of implementation of SISBEN and the affiliation to the SHIR since 1996. BOX 1 provides details and main issues with the application of SISBEN in this program. Table 6. Evolution of Implementation of SISBEN and Affiliation to the Subsidized Health Insurance Regime (SHIR) 1996-2002 (thousands of people) Population 1996 1997 1998 1999 2000 2001 2002 Population 39.281 40.018 40.772 41.539 42.299 43.071 43.834 With SISBEN 14.916 18.568 22.220 24.249 26.578 26.578/a 26.578/a SHIR 5.982 7.027 8.527 9.284 9.510 --- 11.400 SISBEN 1-2/c -- -- -- -- 17.400 17.400/a 17.400/a SHIR/ SISBEN 1-2 (%) -- -- -- -- -- -- 55.7/b /a The same as that of 2000. It is assumed that after the CONPES recommended changes to SISBEN municipalities did not update system. /b According to figures from MSP, about 15 percent of beneficiaries are not in SISBEN databases. Source: DNP, CONPES 055. November 2001, and author's calculations. Box 1. Use of SISBEN By the SHIR: Summary Issues and Experience The health sector reform of 1993 created a contributory health insurance regime (for formal sector workers) and a subsidized health insurance regime for the poor without resources to pay into the contributory regime. Under SHIR regime, each member of a selected family will get a subsidy of about US$67 dollars per year for affiliation to the ARS. The money is not directly paid to the family, but is paid by the municipality to the ARS that has been awarded contracts, on a competitive bidding process. Since 1994 there has been a rapid increase in affiliation to the SHIR to about 11.4 million by mid-2003. Program implementation has been difficult, as it requires coordination of all three government levels. National level determines size of per-capita payments, distributes formula-based transfers to departments and municipalities and provides additional Solidarity Funds to finance the SHIR. Departments contribute with financing using transfers and own funds. Municipal level gets all the funds for the program and contracts out with private health insurance companies the provision of services. Actual provision of services is not done directly by insurers (they are forbidden to do so), but by private and or public hospitals and clinics that have been contracted by insurers. Since the SHIR program was the first and main user of SISBEN, there has been some public confusion about SISBEN and the Program. For many, SISBEN is synonymous with the SHIR. Being this the case, it is important to distinguish between issues related to program design and operation itself, and issues related to the use of SISBEN. Issues related to the program include the great institutional complexity and delays, especially with the transfer of funds. It has been estimated that payment to insurance companies takes more than one year and much more payments to hospitals. The delays occur because central government transfers to departments and municipalities are not always on time, municipalities often have delays of their own for budgeting reasons, and ARS also have some time of their own before they pay hospitals. As a result, many hospitals (public and private) are in great financial difficulties despite the large increase in budgeted expenditure for the program. Recent measures seek to drastically reduce the time resources take to flow to ARS and hospitals, to about 60 days. A major hurdle for the further expansion of the SHIR has been the great difficulties in re-structuring of public hospitals. Public hospitals have pressured health authorities to provide funding directly to finance deficits, rather than sustaining themselves with the sale of services to the ARS. 13Actual health services are not provided by ARS, but by public or private hospitals and clinics that had been contracted by the ARS. 24 Affiliation to the SHIR is made through yearly contracts with the ARS. Many beneficiaries have been renewed their affiliation policies every year. In theory, program beneficiaries should be re-screened every year to assess eligibility; in practice, this has not been done. The problem is that while program eligibility is for only one year, SISBEN is to be updated every three years. The Ministry for Social Protection has recognized this problem and is looking for ways to synchronize program eligibility with updates of SISBEN, especially since the program is so large and difficult to administer by municipalities. This yearly automatic renewal of insurance polities has not allowed the program to exclude some non- deserving people that were included in 1995 at the beginning of the program, and when SISBEN was not fully operational. At the beginning of the program some large cities, such as Bogotá, were allowed to include people using other criteria, such as the Estratification Socieoconomica (ESE), to include people in the program. In 2000, it was estimated that about 15-20 percent of those affiliated to the SHIR had no SISBEN records. However, the fact that not all people in the program have SISBEN records can also indicate manipulation of affiliation by local authorities. Overall, however, as presented below, coverage of people in the bottom 20 and 40 percent of the income distribution with the SHIR has drastically increased in both urban and rural areas to about 48% of the population by 1997, the latest year for which survey data was available for this report. Finally, an important issue affecting the SHIR, and to a lesser extent SISBEN application, is the lack of portability of the SHIR insurance. If a person moves from one municipality to another, she (he) loses the benefit, while she (he) is not automatically affiliated in the arriving municipality. This, results from the transfer system (and the number of places) each municipality gets at the beginning of the year. This lack of portability has been especially problematic for the great number of poor displaced population resulting from the civil conflict. Although special protection is provided to these people, there are a large number that go unattended and that have lost this benefit. Also, many of these people are not registered in the SISBEN survey in the destination municipality. There are two issues regarding the use of SISBEN and the database of beneficiaries. The first is the presence of multi-affiliation (that is to say, people appearing two or more times as beneficiaries of the program). This is the result, to some extent, of misapplication of the SISBEN at the municipal level, application of the SISBEN to the same families in different municipalities, and also technical problems of the software used to input data into the Registry. The absence of a unique ID or social security number makes it hard to check for duplicities and to cross information with other data bases. According to the Superintendency of Health, there were about 7 percent of multi-affiliations in the SHIR and about 3 of duplicates with the data base of the contributory regime, by the end of 2002. Close to one million affiliates, or about 8-9 percent of affiliated, could be eliminated from the SHIR with savings of about 60 million dollars, annually. Finally, the second is the presence of errors of inclusion attributed to the program (about 15-20 percent of those included do not have SISBEN records, and are most probably not the poorest people). Conditional Cash Transfer (CCT-Familias en Acción) This is a program modeled after Oportunidades (Mexican CCT Program) and other similar programs in Latin America, introduced with the financial support of the World Bank (WB) and Inter American Development Bank (IDB), in 2000. The program provides a cash transfer (about US$43 dollars every two months) to selected poor rural families conditional on sending their 0-5 year old children to health check ups and 6-14 years old to schools. The program is a national program administered by a Social Investment Fund of the Presidency of the Republic (beginning in 2003 it will be administered by the Institute of Family Welfare, ICBF). The program selects beneficiaries from the data bases provided by participating poorest municipalities (625). Selected beneficiaries are from SISBEN level 1, who are the poorest population. Payments are made by the program every two months (once health and education conditions have been verified) by the CCT program using electronic transfers to families in all municipalities. The payroll for transfers is prepared using the SISBEN data base complemented with the mother's verified identification number of the citizen card (cédula de ciudadanía) required for the program. By end 2002, the program benefited 315,000 families with over 800,000 children at an annual cost of about US$83 25 million dollars. Once selected, the family stays in the program for three years if they meet established conditions. Without doubt, the program could start rapidly and extent its coverage to over 1.3 million people (including all family members) in less than two years of operation due to the prior existence of SISBEN registries. Major efforts by the program were, however, required to clean databases and update records, in coordination with municipal authorities. Youth Training Program (Jóvenes en Acción) As with the previous program, the Youth Training program was created in 2000 as part of the Safety Net Program (Red de Apoyo Social, RAS) to alleviate the effects of the economic crisis and high youth unemployment. The program provides private- offered training to selected poor youths, 18-25 years old. Youth are selected by the program (centrally) from families with SISBEN levels 1 and 2 SISBEN databases provided by participating municipalities. Youth are given stipends to support them during training and on the job practices. In 2002, over 35,000 youth participated in the program at a total cost of about US$35 million. Applicants´ SISBEN levels are verified against municipal data bases from the seven participating metropolitan areas. Public Works Program The public works program was also a response to the crisis and aimed at increasing employment and incomes of the poor. The program finances municipal and NGO projects identified, prepared and executed at the local level. The program was created in 2000 and is financed by the WB and IDB. In 2003, it proposes to benefit about 67,000 workers and their families in about half the municipalities of Colombia, at a cost of about US$32 million dollars annually. Beneficiaries are selected following a two step progress. First, people from SISBEN levels 1 and 2 are called for to apply to be included in the program in the communities where the projects will be executed and, second, and, because places are usually smaller than the number of applicants, a lottery system is used to select those who will work in the project. The lottery is played by the NGO with active participation of the communities involved. B. Municipal Programs Municipalities have used SISBEN to target a great variety of programs and for other purposes, such as preparation of diagnostics studies of poor areas for their development plans (Table 7). Since the last population census dates back to 1993 (and the results were contested by many municipalities), there is few socio-economic information to inform local plans. Many municipalities have, thus, applied SISBEN surveys to the whole population, and have used SISBEN data for their local plans. In addition to selecting beneficiaries for the SHIR, municipalities have used SISBEN for hospital fee subsidies in public hospitals, education vouchers or scholarships, subsidies for elderly poor, housing subsidies, child care and nutrition programs, among others. Unfortunately, there is little or no aggregate information on local programs, number of beneficiaries attended and budgets spent on those programs. BOX 2 presents how and to what extent SISBEN has been implemented and used in Bogotá, the largest municipality and Capital District of Colombia. 26 Table 7. Uses of SISBEN by Municipalities, 1994-2000 Use of SISBEN Information No. of Municipalities % Development Plan 429 56.1 Sectoral Plans 170 21.8 Employment 126 15.6 Education 403 50.9 Housing 578 74.2 Special programs (Elderly, Children) 573 73.5 Others 131 17.4 No answer 99 9.7 Total 801 /a /a Sum is much higher then 100.0 because there are multiple SISBEN uses. Source: DNP et al. (2001). Box 2. The Implementation and Use of SISBEN in the City of Bogotá Bogotá is the capital city of Colombia and, with a population of about 6.6 million people in 2002, is the largest Colombian city. According to City estimates, of the total population, about 49 percent were poor (as measured by Poverty Line), including about 15 percent extreme poor. Since the launching of SISBEN in 1994, Bogotá has been one of the first to implement and use it (although the degree to which databases have been updated and used has varied by the different 3-year city Administrations). By December 2002, Bogotá's SISBEN database contained about 4.1 million people or over 60 percent of the city's population. About 55 percent of those registered in SISBEN were classified in level 1 and 2, 20 percent in Level 3, while the rest (25 percent) were in levels 4 to 6. There is not an assessment of how old are registries in the data base, but most functionaries indicate that the registry contains questionnaires from 1995 on and even people with two or more filled questionnaires. While levels 1 and 2 are potential beneficiaries for most of the programs, in some special cases, level 3 registries are potential beneficiaries (if judged by a social worker or if they belong to strata 1-2 of the ESE). For national programs operated by the city, such as the SHIR, all have to be from levels 1 and 2. In January 2003, the SISBEN office contracted a massive updating of registries using the revised SISBEN questionnaire mandated by DNP. The target is to have a data base of about 4.2 million people by the end of 2003, which will include all poor people in the city. This new data base is expected to have all records updated and without any duplication problems that are present in the current data base. The city administration has continue using the outreach method by surveying all people in identified poor areas (rather than the application method) for several reasons, including, a) poverty and extreme poverty levels are still high in the city and widespread in the different parts of the city; b) the outreach method ("barrido") is more transparent to poor people and more equitable in that it reaches people with no money for transport and or no time or knowledge for application; c) allows for better quality control of data gathering activities and to better check quality and accuracy of information; and d) the cost of the outreach method per household is much cheaper (see below), allowing for a greater number of people to be interviewed with same administration budget. The application of on-demand surveys remains open for poor not residing in selected areas. Administration of SISBEN. SISBEN is administered by the Under-Secretary for Social Development of the Planning Secretary of the City Government. It has a permanent staff of 9 people including six professional level staff (social sciences, lawyers and information technology) and three assistants. This office is charged with the following responsibilities: (a) planning survey work, identifying areas to be surveyed; (b) taking (contracting) the SISBEN survey and practicing regular updates of the system; (c) solving people complaints about inclusion in the SISBEN system and or complaints with the score points; and (d) distributing the updated SISBEN data base (usually bi-monthly) to other city agencies, DNP and other national programs. Administration and Implementation Costs. The annual personnel costs of the city amount to about US$ 90,000 dollars, including office supplies. The cost per survey varies by urban and rural areas, the latter being about 50 percent more expensive. The per household cost of census (barrido) is about 60 percent of the cost of "on-demand" surveys (Table 8). 27 Table 8. Approximate Total Costs of SISBEN Registry in Bogotá, 2003 Unit Cost Number of Total (US$) Families Costs Urban --Barrido 2.00 1,151,000 2,302,000 --On-demand 3.50 30,000 150,000* Rural --Barrido, census 2.90 9,000 26,100 --On-Demand Not done Total surveys 1,190,000 2,478,100 Audits of contract 200,000** Total Costs 2,678,100 * Includes payments for interviews that can not be materialized for absence of people. ** Includes costs of audits of field work of contractors which are about 8 percent of total. Source: Author's calculations on the basis of information provided by the SISBEN office. Table 9. Estimated Annual Costs of SISBEN Registry in Bogotá, 2003 Definition Costs (US$) Administrative Cost-SISBEN Office --Planning survey work, monitoring, Costs of SISBEN office (permanent 90,000 contracting, audits, city database professional staff plus assistants) Annual Survey Costs One third of costs of registry 893,000 (information is valid for 3 years) Annual Total Costs 983,000 Annual costs per person registered 0.23 Annual cost per benefit Annual costs /111.6 US$ million 0.009 Source: Author's calculations based on information provided by the SISBEN office. Financing. Most financing is provided by the city. DNP proposes to finance surveys of about 720,000 households estimated to correspond to levels 1 and 2, at a much lower rate of about US$1.30 dollars per household. This amounts to US$940,000 dollars or 40 percent of total costs of registry. Programs using SISBEN. The main users have been the Secretaries of Health, Social Welfare (DABS), and Education (Table 10). Table 10. Secretaries and Main Programs Using SISBEN to Select Beneficiaries, 2002 Secretary/Program Targeting Instrument No. of Budget Spent Beneficiaries (US$ Million)* Secretary of Health 84.4 --Subsidized health regime SISBEN 1-2 1,260,000 Secretary of Education --Scholarships SISBEN 1-2 22,000 10.5 Secretary of Social Welfare --Children 0-5 yrs. Old* SISBEN 1-3 & ESE 1-3 38,000 9.3 --Cash transfer to elderly* SISBEN 1-3 & ESE 1-3 22,000 7.4 Total 1,342.000 111.6 *Average exchange rate for 2002 used: 2,500 Colombian pesos per US dollar. Source: Author's calculation based on figures of the different Secretaries. Issues with SISBEN implementation. The first is that the Administration has often complained to DNP about the low cut-off points for levels I and 2, which result in lower numbers of poor than expected by city officials. DNP has argued that the uniform cut-off points guarantee that all Colombians are treated equally no matter where they live. However, the national government provides some freedom to cities to include SISBEN 3 beneficiaries (if lower levels are covered). In addition, in some programs, different agencies have used additional criteria of need to include people from levels 3 of SISBEN. The second is that until 2000, there was not a single agency charged with application and administration of the system. Several secretaries, such as health and social welfare, contracted or applied SISBEN surveys responding to the pressures to increase coverage of their programs. Starting in 2000, the Planning 28 Secretariat is the single agency charged with application and administration of the system. The Secretary of Health has been given power to apply surveys to people who need urgent classification for inclusion into the subsidized health regime or waiving part of hospital fees. These are children less than one year, pregnant women and people suffering from catastrophic health events. Finally, the third relates to several deficiencies in the application software for data entry and processing and updating of the system. The main problems for the city has been the merging and maintaining of a large data base, a process not well supported by the software provided, and the difficulty of updating family records when no household visit is required (birth of a child, death of a person, etc.). Complaints and resolution mechanisms. There have been numerous complaints from people who have not been registered or have higher scores than those habilitating them to receive subsidies. In many other cases complaints are more related to the programs using the system rather than the system itself. The administrative burden on the SISBEN office has, thus, increased considerably by requests from people to be registered or re-interviewed. Many of these complaints have used legal means, such as the Petition Rights or Tutela Rights. In 2002, the SISBEN Office received about 1,000 petition rights and about 150 Tutela Right petitions. As a matter of policy, the SISBEN office will not re-apply surveys if the existing survey is less than one year old. Many of these complaints are expected to be solved with the massive updating of SISBEN started in 2003. Cost-efficiency: The ratio of annual SISBEN costs to the city relative to the annual amounts targeted with the system is 0.009, that is, targeting US$100 dollars costs the City less than one dollar (90 cents). This is the cost of targeting system and does not include the administrative cost of each program. C. Do Programs/Agencies all Use Same (Unified) Eligibility Criteria or Do They Use this Common Database with Different Eligibility Criteria? Most programs use the same eligibility criteria, that is to say, SISBEN levels 1 and 2 or the associated point scores. This has been a source of tension between DNP and some municipalities, specially the larger and richer cities. Some of these municipalities have complained about uniform rules and point scores, arguing that point scores are too low resulting in fewer poor persons in Level 1 and 2 in their cities. DNP has argued that uniform rules provide equal treatment of all Colombians independently of their place of residence. One exception has been the authorization to Bogotá District to include level 3 families at the beginning of the SHIR when few people were registered in levels 1 and 2 (due to under-coverage of SISBEN), and the larger availability of funds for the program in the city. The same eligibility levels apply to all national programs. For departmental and municipal programs, authorities have more freedom to determine cut-off points for their programs. Usually departments and municipalities establish levels 1 and 2 for subsidies and, in some cases, include people from level 3. Level 3 families are eligible for subsidized hospital fees. 29 As illustrated in Table 11, most programs use additional criteria to give priority to special groups within SISBEN levels 1 and 2. Table 11. SISBEN and Other Targeting Criteria Used in Main Programs, 2002 Program SISBEN Priority groups In Practice Subsidized Health SISBEN 1 and 2 Levels. Pregnant women, rural Not known if priority Insurance (SHIR) residents. criteria have been applied by municipalities Low Hospital Fees SISBEN 1: Pays 5% None Applied. ("Vinculados" SISBEN 2: Pays 10%. Program) SISBEN 3: Pays 30%. Conditional Cash SISBEN Level 1 Rural areas (625 Both criteria applied. Transfer (Familias en municipalities smaller Has left some of the Acción) than 100,00 inhabitants poorest out due to and access to Banks for absence of Banks. electronic transfer) Public Works Program SISBEN 1 and 2 Levels Urban areas (500 All criteria applied. (Empleos en Acción) municipalities). Plus, unemployed at time of application to program-- Participate in lottery. Youth Training SISBEN 1 and 2 Levels. 18-25 years old Applied (Jovenes en Acción) Elderly Poor Subsidy SISBEN 1 and 2 Levels. None Applied Source: Criteria established in operation manuals of programs. D. Pros and Cons of Different Types of Criteria Ideally, SISBEN is better used jointly with other criteria to determine not only economic vulnerability, but to make room for considerations of other special circumstances (pregnancy, displacement, for instance) faced by prospective program beneficiaries. In practice, however, this is complex to manage since many of these special circumstances are not measured or weighted in the statistical models, and or the information is not included in the database. If variables are not included in the original database, the information will need to be collected at the time of application to the program. This can give the appearance that the selection process is not fully transparent, and that people are not treated equally. In addition, it is often difficult or impossible to manage new variables in the SISBEN data base, especially when the municipality has a large volume of beneficiaries to register into the program. Currently, it is not possible to register information of new variables into the SISBEN data entry software. The advantage of dealing with means testing for "structural" poverty is that this does not change in short-term and there is no need for periodic updates (say, less than every three or so years) reducing costs and administrative demands. By contrast, dealing with income, employment and other measures of "transient" poverty and need requires periodic updates (even less than every year) increasing costs and putting a heavy burden on administrative capacity of implementation agency and users of the system. The main disadvantage, however, is the failure of the instrument to support safety net programs for the new poor or people affected by special circumstances. There is, thus, a need to develop special targeting instruments to support these programs. 30 E. Process for Households to Appeal, Share of Beneficiary Households that Appeal In its more than nine years of existence, SISBEN has become widely known and recognized as a gateway to many social benefits. Thus, people make every effort to be registered and to be classified in the lowest possible poverty levels. Municipalities, in general, try to keep the system "honest" and strive to have the least possible complaints and re-interviews. As indicated previously in this report, the Colombian 1991 Constitution provides at least two channels for people to complain and appeal if the results of the system are not acceptable to them. The first is the Petition Right (Derecho de Peticion) whereby people can, in writing, ask the municipal SISBEN office to correct or re-take the SISBEN survey. The second is the Tutela Right (Derecho de Tutela) whereby people can ask a Judge to help protect a constitutional right. In SISBEN's case, the most often invoked right is the "right to life" because SISBEN is a pre- requisite to be included in the SHIR.14 The number of complaints varies by municipality, but the most common is the petition right, as seen in Bogotá (See BOX 2, above). 14There have been several landmark rulings by the Supreme Court on the issue of protecting life rights, which have had implications for SISBEN. In fact, in most rulings there has been a confusion between SISBEN and the affiliation to the SHIR. Most people, including judges, understand that SISBEN and affiliation to the SHIR is the same. So, the rulings have considered that SISBEN, by measuring only economic conditions, leaves out other measures of destitution and need, such as ill health, especially of people with AIDS, cancer, or other catastrophic health need. In many cases, courts have ordered affiliation to the SHIR (which is only for SISBEN 1 and 2) even for people with higher SISBEN point scores. 31 6. Evaluations of Effectiveness/Outcomes of SISBEN There has been only one evaluation of SISBEN made in 2000, six years after the initiation of the system, although the government directive mandated revisions and evaluations every three years. Two of the subjects evaluated were, a) targeting accuracy and efficiency of SISBEN, and b) costs of implementation of SISBEN. The study recommended changes to SISBEN welfare Index and variables, and administrative procedures, which are described in Section 7, below. The following is a summary of targeting efficiency of SISBEN and comparisons with targeting efficiency of other targeting instruments used in Colombia (the ESE), and a review of costs of the system. A. Target Accuracy of SISBEN15 For evaluation of targeting accuracy of SISBEN the following steps were followed. The first was to determine a "gold standard" or "true" poverty measure against which to compare predictive values of SISBEN Index. The standard used was predicted income (using consumption for the prediction) divided by deciles of the income distribution. Quintile 5 was the cut-off point to distinguish between poor and non-poor. The second was to establish SISBEN 1 and 2 levels as the poor, subject to program eligibility. Using the 1997 Survey of Living Conditions (ENCV, in Spanish), it was found that 81 percent of those in the 1-5 deciles were correctly classified by SISBEN, while about 69 percent of those classified as poor by SISBEN (Levels 1 and 2) are also in the lower 5 deciles of the income distribution. This means that the error of exclusion (under coverage rate) is about 19 percent while the error of inclusion (leakage) is about 30 percent.16 It is important to note that, by using the survey data to calculate eligibility with SISBEN and comparing that with the standard, is a good way to determine errors of inclusion or exclusion due to the instrument itself (that is prediction errors) since survey data of this sort are not expected to be manipulated by the respondent (as they do not expect benefits resulting from this survey).17 B. Target Accuracy of Programs Using SISBEN There are two recent studies that indicate a good targeting efficiency of SISBEN in the SHIR. The first is the study by Sánchez-Nuñez (1999) which indicated that, while health insurance coverage of people in the first quintile of the income distribution was only 8 percent in 1993, it increased to about 47 percent in 1997, mainly as a result of the increase in affiliation to the SHIR. In rural areas, the increase in affiliation was 15See DNP et al. (2001). 16 The errors of exclusion are likely under-estimated because they only refer to those that were misclassified in levels 3-6, but do not include those that were excluded (not interviewed) from SISBEN altogether. 17 Trying to determine if people (and municipalities) manipulated data to access the SHIR, the MOH conducted a re-interview of 3.108 people affiliated to the SHIR. The results indicated that about 28 percent of people had scores in the new survey higher than level 2, which is the cut-off point to be in the program, and, thus, were leaked into the program. This leakage rate is similar to that found for the instrument itself, and suggests that application of SISBEN has not suffered as much manipulation as reported frequently in the press. 32 even more dramatic from about 2 percent in 1993 to about 46 percent in 1997. Also, the utilization of services by the poorest quintile almost doubled in consultations, and increased by over 25 percent in inpatient care, between 1993 and 1997. Since the subsidized regime has continued to expand rapidly from about 8.5 million in 1997 (when the survey for the incidence study was done) to about 11.4 million by the mid- 2003, coverage levels of the poor have most probably increased even more rapidly. The second is the Evaluation of SISBEN study which presented shares of social spending received by different income deciles, using the 1997 ENCV. The main results of the study (shown in Table 12) are as follows: · The largest shares received by the bottom 20 and 40 percent of the income distribution are in health and social assistance where SISBEN is used more profusely. The share received by the bottom 20 percent was 35 percent, while that of the bottom 40 percent was 65 percent in 1997, mainly the result of the use of SISBEN for the SHIR. · The smallest shares received by the bottom 20 and 40 percent are in public services and housing where SISBEN is not used. In these programs, the main targeting instrument is the ESE which shows poor targeting results. Table 12. Shares of Subsidies Received by Deciles of Income Distribution, 1997 Decile Education Health and Social Public Services Housing Total Subsidy (%) Assistance (%) (%) (%) (%) 1 11 17 10 5 12 2 13 17 9 12 13 3 11 17 10 14 13 4 10 14 11 11 11 5 10 10 11 13 11 6 12 9 12 20 11 7 8 7 11 3 8 8 10 4 11 7 8 9 9 3 8 5 7 10 7 2 8 11 6 Total 100 100 100 100 100 Source: DNP et al. (2001). C. Comparison to Alternative Mechanisms for Targeting As shown in Tables 12 and 13 below, targeting with SISBEN is superior to targeting with the ESE. The share of benefits received by the bottom 20 and 40 percent of the income distribution is the highest and has increased the most in programs that use primarily SISBEN to target subsidies, such as the SHIR (under Health and Social Assistance), and primary and secondary education. By contrast, programs targeted with ESE, such as subsidies to public services and housing, are regressive (that is, the bottom 20 and 40 percent of the population received less than proportional to population), although targeting with ESE (except for water) appears to have improved recently.18 18These comments are only indicative of the relative efficiency of the two systems, as there are two main problems with these comparisons. One is that we are comparing two different types of programs, and it is well known that there are some programs that, by design or their nature, are easier to target than other programs. For instance, housing programs are more difficult to target to the poor because the size of the 33 Table 13. Share of Subsidies Received by the Bottom 20% and 40% of the Colombian Population, 1997 (%) Program / Social Sector Bottom 20% Bottom 40% 1992 1997 1992 1997 Education 23.1 24.2 45.0 45.4 --Primary 39.3 44.9 65.0 73.4 --Secondary 20.6 28.9 47.3 57.3 --Higher 5.1 3.7 14.4 12.2 Health and Social Assistance 29.1 34.8 57.3 65.1 Public Services 17.8 19.5 38.1 40.5 --Water 25.8 18.8 52.8 39.1 --Electricity 15.6 19.8 34.1 41.3 Housing 14.5 16.8 38.9 41.6 Source: DNP et al. (2001), based on data from Velez (1996) for 1992 and calculations with 1997 survey. D. Costs and Cost Efficiency of SISBEN Although DNP produces costs benchmarks for municipalities to follow, there is no way to know municipalities´ actual costs of SISBEN administration and implementation. The following are cost estimates of SISBEN design and implementation on the basis of DNP benchmarks, and the experience of some cities, such as Bogotá, corresponding to the new application that started in 2003. According to Table 14, total national SISBEN costs are estimated at about US$18.3 million, or about US$2.3 dollars per household (about 8 million households), or US$0.57 dollar per person registered (a little more than 32 million people) in the 1050 Colombian municipalities. transfer is typically large and attracts many middle-income people. The second is that to determine the net impact of a targeting instrument we need a counterfactual--that is to say, what would be the targeting outcome in the absence of SISBEN, given that there was a political decision to target social spending. Random allocation is most likely not a proper counterfactual because no government official or Congress would approve such an allocation method. An interesting evaluation of geographic targeting versus household targeting with proxy-means test was done for the Social Protection Program (Red de Protección Social) of Nicaragua (IFPRI, 2002). For this evaluation, a sample of census areas were selected to be beneficiaries of the program (on the basis of poverty maps which combined survey and census information), and another sample of census areas were selected in which individual household were selected with predicted poverty levels. One of the results is that individual targeting is better when program budgets are larger allowing the program to cover not so poor areas. Also, the marginal costs of applying proxy-means testing, as opposed to geographical targeting, are not so great (only about 30 percent higher), as in both situations a census form needs to be applied to households--in the first case a larger questionnaire to include predictor variables. For geographic targeting, an incorporation form needs to be filled to all households. 34 Table 14. Approximate National Costs of SISBEN Registry, 2003 Area Unit Cost (US$)* Number of Total Costs Households (US$) Urban --Barrido-census of poor areas 1.80 5,200,000 9,360,000 Rural --Barrido-census of poor areas 2.70 2,800,000 7,560,000 Total surveys 8.000,000 16,920.000 Audits & Other** 1,350,000 Total Costs 18,270,000 Costs per household 2.3 Costs per person registered 0.57 * Exchange rate for 2003 estimated at 2,700 pesos per US dollar. ** Estimated at about 8 percent of total expenses. When municipalities contract out survey work, they need to pay independent audits of contracts. Source: Author's calculations on the basis of information provided by the SISBEN office at DNP. Table 15 presents annual costs of registry and the administrative costs which are necessary to run and use the system. Since registry information is valid for three years, according to DNP guidelines, total registry costs are divided by three to get annual data. Administrative costs include, a) costs of DNP for design, monitoring application, planned random audits to databases, and management and administration of the central data base, and; b) estimated administrative costs of municipalities which include the costs of SISBEN administrators and staff, supplies, costs of on-demand surveys (assumed to be small given the large size of the population to be covered in the initial outreach survey), and the costs of appeals, and management and operation of the data base. Based on the costs of Bogotá, these administrative costs are estimated at about 8 percent of the cost of the registry. Annual total costs per person registered amount to about US$ 21 cents, while costs per beneficiary amount to about 52 cents. Finally, the cost of targeting 100 dollars with SISBEN cost less than one dollar (about 70 cents), on average, nationally. These low costs are due to the inexpensive application of SISBEN and, to a large extent, to the extensive use of SISBEN by many programs in Colombia. About US$940 million dollars (about 1.1 percent of GDP) are targeted with SISBEN in more than five large national programs operated jointly or with participation of municipalities. Table 15. Estimated Annual Costs of SISBEN Application Nationally, 2003 Costs Items Description Costs (US$) National Level-DNP --Design, monitoring, audits, central 7 permanent professional staff + 1 95,000 database assistant secretary Municipalities --Administrative costs* Cost of SISBEN office, on-demand 487,000 surveys, appeals, etc. --Survey costs One third of costs of registry 6,090,000 (information is valid for 3 years) Total annual costs 6,672,000 Annual costs per person registered Total costs / 32 million (2003) 0.21 Annual costs per beneficiary Total costs/ 12.9 million (Table 5) 0.52 Annual costs per benefits ** Total costs / US$941 million (Table 5) 0.007 * Estimated at about 8 percent of costs of surveys. ** Does not include beneficiaries (spending) of departmental and municipal own programs. Source: Author's calculations based on information provided by DNP and Bogotá City. 35 7. Recent Update of System Based on the SISBEN study, there were recommendations to update the system in two fronts. The first was in the determination of new variables (and weights) to predict better the situation of the poor given changes occurred in economic and social situation of Colombia since 1993. The second was to improve application procedures, including survey application, data audits and data management at local and central (DNP) levels. In the new SISBEN application there will be a central data base managed by DNP to be shared with local and national programs. The changes were designed in 2001-2002 and application of new questionnaire and procedures started in January 2003, nationwide. As in the original SISBEN, there will be a massive application of the questionnaire in selected poor areas identified by the cities and municipalities, to be followed by application on-demand for those not included in the initial survey. New variables, weights and questionnaire For the determination of new variables and weights, DNP used the 1997 ENCV, which is representative of the country (rural-urban and major regions). The objective was to determine which SISBEN variables are still relevant (i.e., able to discriminate among people by welfare level), which relevant variables had been omitted in the previous SISBEN index, and which variables (relevant) are subject to manipulation by respondents or applicants and other members of the process (local authorities) and, thus, should be eliminated and or reformulated in the new questionnaire. Determination of relevant variables was done using principal component analysis for qualitative variables, described earlier in this report. The identification of variables which were subject to manipulation was pursued by comparing the frequencies of some variables of households with SISBEN levels 1 and 2 in the 1997 ENCV (where no manipulation is expected), with results of actual application of the SISBEN questionnaires, using a consolidated database of over 600 municipalities (DNP et al. (2001)). The results indicate that correlations between estimated proportions of sub-standard wall materials, floor materials and waste disposal with the ENCV and aggregate SISBEN data base for SISBEN levels 1 and 2 were very high (over 95 percent in urban and rural areas) indicating that responses to these variables in SISBEN most likely represent the real situation.19 This is expected since this information is taken in the field during home visits. Other frequencies of variables were not compared as the definitions varied among the two information sources. Variables of the Old SISBEN Index which were recalculated or excluded in new Index There were a number of variables in the old SISBEN Index that had lost their discriminatory power or that were eliminated because they were not properly measured or calculated, in the new SISBEN Index. These were: wall and roof materials, availability of potable water, solid waste disposal (changed to whether family has access to collection of solid waste), average school years of those over 12 years of age 19There may be some problems with these comparisons since not all poor people are surveyed in the SISBEN in all municipalities, and the proportion of poor people surveyed may be quite different in the municipalities. 36 (eliminated because it was highly influenced by schooling of those still in school), school years of highest income earner (changed to school years of family head), affiliation of social security of highest income earner (changed to affiliation of family head), presence of durable assets (changed to number of selected durable assets), proportion of employed people in family (eliminated due to the little reliability of information). The income variable was also eliminated because it is highly unreliable and the predictive power of the new SISBEN Index did not improve with the inclusion of this variable. Variables and weights of the new SISBEN Index The new SISBEN index (as compared with the old one) is calculated on the basis of, a) a more precise definition of variables (for instance, in the old Index, the education variable was measured as the average years of schooling of the older than 12, while the new education variable is education of the head of household), and better description of quality of service (availability of water service at home, instead of simple availability of water); and b) introducing some variables which are different between urban and rural areas. While there are only a few such variables, the old SISBEN Index made no distinction. The new variables are presented in Table 16. Table 16. Variables Included in SISBEN Index and Questionnaire for Urban and Rural Areas (2003) Variables Urban Areas Rural Areas Geographic location of Region, size of urban center, urban Region, size of urban center, area dwelling unit strata level, risk area/a (small town or dispersed population) Housing conditions Floor materials, location of wager Floor and wall materials, location of (quality and comfort) service (in house, outside), location water service (in house, outside), of sanitary unit, number of sanitary location of sanitary unit, number of units, availability of sanitary service sanitary units, availability of sanitary with shower. service with shower. Services available to Collection of solid waste, Collection of solid waste, telephone, dwelling unit telephone, combustible materials combustible materials for cooking, for cooking availability of electricity, type of electric connections. Schooling and school Years of schooling of head and Years of schooling of head and attendance spouse, proportion of children 5-11 spouse, proportion of children 5-11 out of primary school, proportion of out of primary school, proportion of children 12-17 out of secondary children 12-17 out of secondary school. school. Demographic Number of family units (hogares) Number of people per living room characteristics living in dwelling unit, age of head, (hacinamiento), age of head, number number of children less than 6 years of children less than 6 years of age, of age, gender of head, whether gender of head, whether single single parent, presence of parent, presence of handicapped handicapped persons unable to persons unable to work and without work and without any income. any income. Affiliation to social Number of people affiliated to Number of people affiliated to security system. contributory system (health, contributory system (health, pensions). pensions). Durable goods Number of durable goods within Number of durable goods within refrigerator, washing machine, refrigerator, washing machine, color cable TV, color TV, water heating TV, water heating system, oven and system, oven and air conditioning. air conditioning. /aRefers to the socioeconomic strata obtained by the ESE to target public services. Source: CONPES SOCIAL 055, November 2001. 37 Structural poverty versus transient poverty in the SISBEN Index As in the old SISBEN Index, variables and weights measure "structural" poverty which changes very little with economic cycle, since occupation and income variables contributed little in the old Index, and were dropped in the new one, as they were presumed to be highly manipulated and there were not independent means of verification. Although, the issue of not being able to deal with "transient" poverty and risks associated with the economic cycle or family risks have been amply discussed, the decision has been to stay with "structural" variables which are more easily verifiable and harder to manipulate by respondents, municipal authorities and or survey takers. Also, these variables change little in the short time (say, less than three years time) reducing the need for updates, which are costly and highly demanding in administrative capacity of municipalities. Thus, while SISBEN may be the appropriate instrument to target programs such as the SHIR, the conditional cash transfer (CCT-Familias), housing subsidies, subsidies to elderly poor, and others for the long-term poor, it is much less adequate for targeting other programs, such as public works, programs for displaced populations, programs for pregnant women and youth, which respond to people affected by economic cycle and or special circumstances. In these cases, other information (in addition to SISBEN's) is needed. Moreover, even in the case of the SHIR, the use of SISBEN has been challenged by the High Constitutional Court, on the grounds that the SISBEN Index recognizes no special health needs or vulnerabilities of the sick (especially with AIDS and other catastrophic events), but considers only economic destitution. These rulings have obliged municipalities to include in SISBEN registries, and in the SHIR, some families whose point scores were higher than the cut-off points for level 2. These are, however, a relatively small number of cases, but health insurance authorities caution that if this becomes common practice, the SHIR would be subject to adverse selection risks, as people press for inclusion into the system once they become sick. 38 8. Summary and Recommendations The following conclusions can be derived from this report: · Proxy means testing systems such as SISBEN are more appropriate for use in targeting programs for the chronic (structural) poor than the transient poor. This has the advantage that updating is not needed as frequently, but the disadvantage of not being able to "catch" vulnerable families when they fall into poverty. Pure income measures, on the other hand, are more able to measure short-term (transient) changes in welfare, but suffer the drawbacks of (a) requiring updating much more often; and (b) being unreliable without verifications of incomes and employment, which are difficult in developing countries due to limited administrative capacity and the large informal labor. Due to such difficulties, the income variables are no longer weighted in the new SISBEN system. · Some important deficiencies in the SISBEN system include: (a) lack of central level (DNP) monitoring of application procedures; (b) lack of auditing of databases; (c) infrequent evaluations; and (d) lack of consolidation into a central database. Some of these deficiencies are being addressed in the new application of SISBEN, but it remains to be seen how DNP can monitor application of SISBEN in the over 1,050 municipalities, and there are no plans for future evaluations of SISBEN. Also, the problem that not everybody in Colombia has a unique identification number (cédula de ciudadanía or registration number for new born), could be a key stumbling block in SISBEN for the purposes of (a) detecting duplications in SISBEN databases within and among municipalities; (b) compiling a central database from the local databases; (c) updating; and (d) auditing of databases. The issue of updating needs to be clarified in program manuals and directives. These should clearly spell out what information should be updated in SISBEN offices (demographic, etc.) and which need a home visit (change of address, etc.) and when. The decision to use a common software and distribute this to all municipalities was a key important decision, but the original software presented problems when updating records, merging databases and managing databases of large cities. These problems need to be corrected in the software for the new SISBEN application. · Despite these problems with SISBEN, targeting has substantially improved in Colombia in the last few years. While it is important to highlight the strong normative and political decision to target social spending after the 1991 Constitution, a great part of the improvement can be traced to SISBEN. Programs that use SISBEN show the largest gains in targeting. In the SHIR coverage of the bottom 20 percent of income distribution increased from about 2 percent (8 percent urban) in 1993, to over 47 percent in 1997, the latest year for which information is available. Given the large increase in the SHIR program since 1997, coverage is expected to have increased significantly more for this group in recent years. As a result, the share of health subsidies received by the bottom 20 and 40 percent increased from about 29 and 57 percent in 1993, to 35 and 65 percent respectively in 1997. 39 By contrast, programs using the geographic targeting instrument, ESE, such as, subsidies for public services, mainly water and electricity, have been regressive or proportional to the population. · Annual costs of targeting with SISBEN have been small in absolute terms and relative to the amounts of benefit transferred. SISBEN's annual costs are about US 6.7 million dollars, or US$0.21 per person registered, or US$0.52 per beneficiary. Conservative figures indicate that targeting US$100 dollars of benefit costs about US$70 cents. Cost per dollar spent has been reduced drastically as many national and local programs have used SISBEN to select their beneficiaries. Although many programs use other criteria to assess urgency or need for the specific program being targeted, most apply those criteria to people within SISBEN welfare levels 1 and 2. · Implementation of SISBEN by municipalities has been a relatively speedy process despite initial delays occasioned by financing difficulties. By 1996, two and half years after launching of the system, most municipalities had implemented SISBEN despite their great disparities in administrative and financial capacity. By 2000, most municipalities had built the institutional capacity to plan, do the surveys, and run the system. Most had created SISBEN administration committees, appointed a SISBEN administrator, and provided office space and computational equipment. The new SISBEN application currently underway is expected to be made in only one year, covering about 8 million households nationwide. · SISBEN has survived three different national and local administrations and, despite its problems, is appreciated by the public and program administrators. While anecdotal evidence suggests manipulation and political interference, overall, SISBEN has been perceived as providing some objective and uniform classification criteria to municipal authorities, so that they can exercise their constitutional mandate of targeting social spending to the poor and most vulnerable. There is, however, much room for improvement: · SISBEN has not been updated regularly (every three years) as indicated by normative acts and operating manuals, although the situation varies greatly by municipality. This results from inadequate funding by DNP and municipalities and unclear updating procedures. Law 715 of 2001 mandates municipalities to provide funding for updates but this needs to be closely monitored by DNP to ensure proper annual allocations for this purpose. Also, since SISBEN is heavily used by national programs, DNP needs to consider providing more funding for its application (currently provides about 40 percent of total costs). · The quality of data needs to be improved significantly. As shown in the evaluation of SISBEN study, there are problems with the dates of the survey which signal greater problems of application and use of the software. So far, nobody has audited municipal databases, and there is no way that the central government can certify that municipalities are following stated 40 procedures. These problems are expected to be corrected with the new application of SISBEN which considers auditing and certification by DNP. It remains to be seen how DNP can do this activity for the over 1,050 municipalities countrywide. · Build a central data base (CDB) of potential beneficiaries of social programs (SISBEN Levels 1-3). This is critical for several reasons. First, in addition to facilitating the elimination of duplicates, it permits cross-checking of program data bases with the CDB to make sure that people are registered, and that beneficiaries have the same poverty level--in program and in CDB. Currently, it has been estimated that about 15-20 percent of people in the SHIR do not have SISBEN surveys and scores. Second, make sure that priority groups are given priority by mayors to program affiliation (for instance, priority groups include Level 1, children, pregnant women). Third, cross-checking with other data bases such as that of the contributory health insurance system, financial system, cadastral records, etc. Fourth, facilitate portability of benefits across municipalities. A central data base allows a person to be identified no matter where his (her) place of residence is, and retains a benefit if he (she) qualifies for it. · Establish formal channels for sharing information. So far, municipalities are completely autonomous for management and use of information. In the new application of SISBEN, municipalities have to report and send databases to DNP for their review and auditing. Much needs to be clarified as to: (i) how DNP can share databases with other Ministries and Social Agencies (such as the ICBF, National Training Center, SENA), municipalities and the public at large: (ii) how often, and by which means, municipalities can update the CDB regularly with information of new registries (applied on demand), updates for change of family composition or change of address, (iii) how the CDB can be updated with program information from the different agencies. That is to say, if a person or family in the CDB gets a benefit, how the agency providing it reports to the CDB. · Establish and apply penalties for manipulation and misuse. A system so critical to provide benefits, and so potentially vulnerable to manipulation by the different actors including municipalities, local leaders and beneficiaries, requires the enactment of a strong set of conduct rules that need to be monitored and penalized for misuse or manipulation. The new SISBEN application system and accompanying decree, established penalties for municipal workers and authorities for manipulation and or misuse, and for beneficiaries who do not report the correct information. It remains to be seen whether these penalties will be applied or are enough to deter misconduct and illegal behavior. 41 References Castaño, Elkin. (1995). Metodología para la predicción del indicador de pobreza SISBEN en términos de las variables observadas en el censo de 1993. Mimeo, DNP-Misión Social. Congreso de la Republica. Law 60 of December 1993. Congreso de la Republica. Law 100 of December 1993. Departamento Nacional de Planeación, Ministerio de Salud, Misión Social, PNUD. (2001). "Evaluación Integral del SISBEN". Departamento Nacional de Planeación. Dirección de Desarrollo Social, Grupo de Calidad de Vida. (2002). Manuals from 1 to 7 of New SISBEN. Departamento Nacional de Planeación, Unidad de Desarrollo Social, Misión Social. (1994). Manuals from 1 to 5 of SISBEN. Departamento Nacional de Planeación, UDS-Misión Social. (January 21, 1994). CONPES No. 22: "Focalización del Gasto Social en las Entidades Territoriales". CONPES Social No. 055 (November 22, 2001). "Reforma del Sistema de Focalización Individual del Gasto Social". Hentschel, J., J. Lanjouw, P. Lanjouw y J. Poggi. (2000). "Combining Census and Survey Data to Trace the Spatial Dimension of Poverty: A Case Study of Ecuador". The World Bank Economic Review, 14 (1): 147.65. International Food Policy Research Institute (IFPRI) (2002). "Sistema de Evaluación de la Fase Piloto de la Red de Protección Social de Nicaragua: Evaluación de Focalización". Managua, Nicaragua. Ministerio de Salud (2000). "Evaluación del SISBEN: Eficiencia y Eficacia Institucional del Proceso de Identificación, Clasificación y Selección de Beneficiarios". Informe final. Unión Temporal BDO Salud AGE-CCRP. Sanchez Fabio y Nuñez Jairo. (1999). "Descentralización Pobreza y Acceso a los Servicios Sociales. ¿Quien se Benefició del Gasto Público Social en los Noventa? Coyuntura Social. Velez, Carlos Eduardo, Castaño Elkin and Deutsch Ruthanne. (1998). An Economic Interpretation of Colombia's SISBEN: A Composite Welfare Index Derived from the Optimal Scaling Algorithm. Velez Carlos Eduardo, DNP, Misión Social. (1996). "Gasto Social y Desigualdad: Logros y Extravios". Marzo. 42 Annex 1. Legal Directives for Targeting and or Use of SISBEN Article 357 of National Constitution. Establishes rules for distribution of social spending to departments and municipalities. It establishes that 60 percent of that spending be distributed in direct proportion to the number of people with unsatisfied basic needs (NBI index). Article 30 of Law 60 of 1993. Law 60 of 1993 defines targeting and provides formula for distribution of resources for social spending in territorial entities to be applied since 1994. CONPES No. 22 DNP-UDS-Misión Social: "Focalización del Gasto Social en las Entidades Territoriales" (January 21, 1994). A CONPES is an authoritative statement of government policy and guidelines issued by the Council of Ministers for Social and Economic Policy. This CONPES established that municipalities must target resources for education, health, housing and potable water to poor and vulnerable people. It defined demand subsidies, as resources given directly to beneficiaries to help them have access to basic services. CONPES No. 40 (September, 1997) This CONPES supported the targeting criteria established in 1994 and mandated the MOH and DNP to do an evaluation of SISBEN application. CONPES Social No. 055 (November 22, 2001). "Reforma del Sistema de Focalización Individual del Gasto Social" This CONPES established the adjustments and changes required in order to improve the SISBEN on the basis of the results of the evaluation of SISBEN study carried out in 2001. Article 94 of Law 715 of 2001. Defines general elements for targeting of social spending and gives recommendations to social CONPES to set out every three years criterias for determination, identification and selection of potential beneficiaries of social programs. Also, mandates municipalities to allocate resources for updates of SISBEN surveys. 43 Annex 2. Summary Content of SISBEN Operating Manuals. The SISBEN has 7 manuals to cover all aspects of implementation, management and maintenance of the system, as follows: Manual No. 1 Administration Presents concepts and basic instruments of targeting of social spending and describes administrative activities required to implement and operate SISBEN. It includes the way to obtain information, the human resources needed, budget, geographical information, and the way to keep information updated. Manual No. 2 Interviewer Describes procedures, rules, norms, concepts and definitions that the interviewer must manage in filling the questionnaire. Manual No. 3 Directives for gathering and capturing of information. Describes procedures and activities to follow in the process of gathering and inputting information. It also describes quality control activities. Manual No. 4 Supervisor Includes norms, functions and procedures that allow supervisors to verify consistency of data, and to detect range errors, validate information and to administer the data gathering task. Manual No. 5 User Includes procedures to manage inputting software and activities to carry out administration and use of information. Manual No. 6 Data Quality Control Includes procedures to manage data quality control software and subprograms to input random samples, verify questionnaires and provide comparisons with SISBEN data bases. Manual No. 7 Normativity Describes legal framework of SISBEN. Describes responsibilities of agents involved in process of administration, operation, updating, use and management of SISBEN. Describes unaccepted behavior related with management, operation and or use of SISBEN and penalties for those involved. 44 Annex 3. Variables and Weights of Statistical Principal Component Model Table 3A describes the factors, categories and variables used in the construction of the SISBEN index and scores assigned to each variable. Table No. 3A Variables and Weights in SISBEN Index, Urban and Rural* Description of Categories and Score** Variables Urban Rural Human Capital, Social Security and Firm Size Factor Schooling of oldest wage earner No education 0 0 Some elementary 1.6239 0.2528 Complete elementary 3.4435 1.5723 Some secondary 5.0039 3.7663 Complete secondary 7.3434 5.1876 Some of higher education 9.7833 11.4263 Complete higher education 11.5460 13.0990 Graduate studies 12.4806 18.3860 Mean schooling for people 12 years old and older 0 0 0 4 1.6570 0.3269 5 2.9947 1.5793 10 4.9690 3.2931 11 7.6387 6.3560 15 9.4425 12.3633 16 10.6900 12.5863 More than 16 11.1396 18.0414 Social security and plant size Without social security and works 0 0 alone or doesn't work Without social security and works 1.1660 0 in plant with 2 to 9 employees Without social security and works 2.6545 1.4320 in plant with 10 or more employees With social security and works 3.9539 2.6097 alone or doesn't work With social security and works in a 5.8427 3.6514 plant with 2 to 9 employees With social security and works in a 6.9718 4.5259 plant with 10 or more employees Demographic, unemployment, income and room crowding Proportion of children below 6 years old Greater than 0.65 0 0 From 0 to 0.65 0.2237 0.2181 Cero 1.4761 1.1626 Proportion of family members employed Less than 0.30 0 0 From 0.30 to 0.60 0.6717 1.0806 From 0.60 to 0.90 1.7390 1.8668 Greater than 0.90 4.0149 3.1957 Room crowding (number of rooms per person) Urban Less than 0.20 0 0.20 to 0.30 0.5584 0.30 to 0.40 1.6535 0.40 to 0.70 2.5727 0.70 to 1.00 4.3886 1.00 to 4.00 6.0042 45 Description of Categories and Score** Variables Urban Rural Greater than 4.00 8.3828 Room crowding (number of rooms per person) Rural Less than 0.30 0 0.30 to 0.60 0.8956 0.60 to 1.00 1.8988 1.00 to 4.00 2.9379 Greater than 4.00 4.9313 Per capita income in minimum wage units Less than 0.15 0 0 From 0.15 to 0.25 0.8476 1.1079 From 0.25 to 0.35 2.1828 1.9561 From 0.35 to 0.50 3.5362 2.9685 From 0.50 to 0.75 5.3636 3.9781 From 0.75 to 1.00 7.0827 4.9210 From 1.00 to 1.25 8.2489 5.6862 From 1.25 to 1.50 9.4853 5.6862 From 1.50 to 2.00 10.2098 5.6862 From 2.00 to 3.00 11.3999 7.7840 From 3.00 to 4.00 13.0872 8.5781 Greater than 4.00 13.7378 9.3504 Housing Characteristics and Appliances Factor Wall material Without walls, bamboo or other 0 0 organic materials Zinc, cloth, cardboard, cans 0.2473 3.2042 Raw wood 2.0207 4.5588 "Bahareque" 4.8586 3.4319 Adobe, "Tapia pisada" 6.2845 3.4319 Block, bricks, stone, prefabricated 7.7321 7.0780 material, polished wood Predominating roof material Straw or palm leaves 0 0 "desechos" (cardboard, cans, 2.1043 1.1312 "sacos", etc) Zinc, asbestos, cement, without 3.7779 3.7615 "cielo raso" Clay tile, zinc, asbestos, cement, 5.0973 4.8771 with "cielo raso" Predominating floor materials Dirt 0 0 Burda wood, boards 2.9037 2.4628 Cement 3.6967 3.7474 Floor tile (clay, vinilo), brick or 5.8712 5.4726 paving tile Wall to wall carpet, marble polished 6.8915 5.7495 wood Home appliances No basics 0 0 Up to three basics 2.1435 1.6865 Four basics without washer 3.0763 - Up to three or four basics with 4.7194 2.7080 laundry machine Public utility services factor Water supply system River or spring 0 0 Public fountain 1.1601 1.0523 46 Description of Categories and Score** Variables Urban Rural Well without water pump, container 2.6497 1.6591 or rain water Well with water pump 4.6037 2.2640 Container truck 6.1693 3.5759 Aqueduct 7.2554 7.2438 Sewage disposal No sewage 0 0 Latrine 2.4519 1.4398 Toilet without conection to sewer or 3.3323 3.0718 septic tank Toilet with conection to septic tank 3.9615 4.2110 Toilet with conection to sewer 6.8306 7.3137 Garbage disposal Yard, lot, river, etc. 0 0 Local container or public trashcan 2.1291 1.5414 Picked up by public services 3.2701 2.6758 * The variables and weights are those of the original (1994) SISBEN Index. The new Index is currently being calculated following same methodology, as described in text (Section 8). ** Summary of weights and scales after typification. Sum of maximum scores per variable is equal to 100. Source: Velez et al. (1998). 47