64379 Bertha Briceño This is the second note in a monthly series on government monitoring and evaluation (M&E) systems led by the PREM Poverty Reduction and Equity Group under the guidance of Jaime Saavedra, Gladys Lopez-Acevedo, and Keith Mackay, with contributions from several World Bank colleagues. The main purpose of this series is to synthesize existing knowledge about M&E systems and to document new knowledge on M&E systems that may not yet be well understood. The series targets World Bank, donor staff who are working to support client governments in strengthening their M&E systems, as well as government officials interested in learning about the uses and bene�ts of M&E and in adopting a more systematic approach toward M&E in their governments. This note characterizes various types of monitor- later draw lessons relevant for the design of new ing and evaluation (M&E) systems, including M&E units. Development banks, bilateral aid examples from three government-based systems agencies, independent organizations, academia, in Latin America: Mexico, Colombia, and Chile. and, overall, governments, are closely involved The characterization outlined here ranges from in the M&E of development programs. Develop- completely outside independent bodies, to ment banks and bilateral aid agencies often use strongly centralized government systems, to highly M&E to measure development effectiveness, be decentralized systems. Each type of system has accountable to donors and stakeholders, and to different advantages that result from the interplay demonstrate transparency. Academics, typically of demand and supply forces: on the supply side, through centers affiliated with economics and actors produce M&E information for a variety of public policy faculties, conduct very rigorous stud- intended purposes, the demand responds with ies with a knowledge-generation focus in the �eld actual utilization, revealing the real incentives of development economics. Academics also often of the systems’ clients. In addition, strategies are intend to influence policy, although less directly presented to mitigate potential disadvantages of than governments. the different types of systems. In general, governments carry out signi�cant As countries are increasingly expressing a M&E activities as a way to increase program effec- need for M&E support, this analysis should help tiveness and improve resource allocation. Govern- champions and task team leaders identify suit- ment-based systems typically serve internal clients able opportunities for their particular context; from the executive branch, such as planning and identify and understand the forces that drive the budget offices; central units within ministries or creation of new M&E units and bodies; and better agencies in charge of planning and budget; and, in understand the tradeoffs involved. some cases, special evaluation units. All of these government units commonly use the M&E system as a tool to improve the efficiency and effectiveness of resource allocation, with a sort of centralized control approach. A good starting point is to ask who conducts Three government-based systems in Latin- M&E activities, and for what purpose, in order to America, CONEVAL from Mexico, SINERGIA from Colombia, and DIPRES from Chile are Independent, external body: transparency presented as examples (see box 1). and social control emphasis Internal clients also include executing or When transparency, social control, and account- implementing agencies in governments and task ability are the main drivers of M&E activities team leaders in development banks. They are and there is a strong emphasis on independence, generally more interested in revising processes, creation of outside independent bodies is appeal- changing and improving practices with a results ing. Location outside the government and inde- orientation, generating visibility for the projects, pendent �nancing provide for the possibility of and responding to constituencies and managers freely making assessments without any improper with concrete information. influence. The M&E body enjoys a high degree of Efficiency and effectiveness of resource al- independence that ultimately reinforces external location are also main focuses of internal clients credibility. from bilateral aid agencies and development Examples of these types of M&E bodies banks, such as boards of donors and directors, include donor and civil society initiatives, inde- internal evaluation units, and sector/regional pendent oversight institutions, and think tanks. managers. Some examples of bodies that fund or carry out In terms of external clients of government- M&E activities on their own are Transparency In- related M&E systems, there is potential for use by ternational, the International Initiative for Impact congress and civil society, which are interested in Evaluation (3ie), the Center for Global Develop- fostering transparency, accountability, and social ment, and the Bogotá Cómo Vamos initiative in control.1 On the other hand, M&E activities that Colombia. The disadvantage of an external body development banks and donors carry out have is that it may lack adequate access to informa- as external clients policy makers, client govern- tion, insight, and contextual knowledge. These ments, legislative bodies, and constituencies in bodies often rely on building a long reputation of general. Table 1 presents a simple classi�cation independence and nonpartisanship to influence of main clients and their corresponding usage policy making. Transparency and accountability focus. use might be easier for an independent M&E body. However, influencing policy remains a signi�cant challenge for such bodies (whether in academia or donor-�nanced entities). The majority of the M&E bodies for development programs strive for independence, transparency, Government-owned M&E autonomy, credibility, usefulness and ownership, system in a centralized fashion: among other qualities.2 management or control tool In theory, an ideal M&E system should be When the M&E system is housed in the govern- independent enough to be externally credible and ment’s center (budget, central authority, planning, socially legitimate, but not so independent to lose presidency, or internal control office), it usually its relevance. The M&E system should be able to serves as a tool for management or for budget con- influence policy making by ensuring adoption of trol. Under central budget authorities, M&E is recommendations informed by lessons learned, often used to control efficiency and/or efficacy and it should be sustainable over time and through of resources, and as such, the M&E body usually government transitions, because this system is enjoys considerably more power to enforce the rec- perceived as responsive to the needs of clients and ommendations resulting from M&E assessments. useful to its main stakeholders. In the extreme enforcement version the M&E The M&E bodies are different in their nature body has direct power over the evaluation agenda and predominant incentives: institutional ar- and enjoys a prominent position and support from rangements range from independent external bod- congress. This is the case of DIPRES in Chile. Loca- ies, to strongly centralized government systems tion under the budget authority presumably also under budget authorities, to highly decentralized provides better integration of M&E into the budget- government system. ing and executing stages of the public policy cycle. On the other hand, when M&E activities are tunities for best practice sharing; and the need mainly intended to serve as managerial tool, a for maintaining its own technical expertise. Some coordinating central body can be placed outside may argue that the separation between evaluator direct budget authorities to avoid a force-fed ap- and executor is less clear, which may increase the proach. The body seeks managerial buy-in and potential for undue interference, but in theory ownership and expects voluntary adoption of recom- this should not be the case if the use is mainly mendations. For this, the M&E body invests highly internal and the true motivation is improving in consultations, in demonstrating the bene�ts self-performance rather than reporting. of evaluation as a managerial tool, in capacity- building activities, and in establishing a favor- Supporting strategies able cultural climate for M&E. Both Colombia’s Some of the potential risks of the different types of SINERGIA and Mexico’s CONEVAL regularly M&E systems could be mitigated through various carry out capacity-building and dissemination supporting strategies or complementary institu- activities to promote a results-oriented manage- tions, such as the following: ment culture. Key alliances. In the government systems Some of the advantages of the centralized analyzed, support from congress, fluid communica- M&E agency include more authority to prioritize tion, and promotion of alliances with government key programs to be monitored and evaluated central authorities are common strategies used when resources are limited; the ability to pursue to overcome weak enforcement powers and to recommendations that affect multiple agencies promote adoption of recommendations in reluc- or the whole government; the ability to pursue tant sectors. Examples of these alliances include standard practices, best practices, and quality CONEVAL with the Ministry of Finance and SIN- methodologies; the possibility of maintaining ERGIA with the Presidency Office. In Organisation specialized technical staff for support (which for Economic Co-operation and Development individual agencies may not be able to afford countries, such as Canada, Australia, the United especially in the beginning when the volume of States, and the Netherlands, the support provided M&E activities is low); more opportunities for by the national audit offices has been considerable: generating synergies for learning and economies they have helped by reviewing (via efficiency au- of scale in data collection; reducing duplication of dits) the planning, conduct, and use of M&E in the efforts; and maintaining an institutional memory executive government; by publishing good prac- of M&E practices. tice guides on these issues; by analyzing the quality of department/agency data and data systems; and Government-owned M&E decentralized by using their influential and prominent position system: full ownership by agencies to advocate the merits of M&E. Alliances with In the extreme decentralized version, the agency national statistical offices, although not pursued so that runs the program also conducts the M&E far, could become important as well. activities to improve its own performance, rather Financial independence. The ability of the than reporting to a central entity or authority. The coordinating agency to influence policy and adop- advantages are in-depth contextual knowledge and tion of recommendations as well as set an evalu- better access to data and disclosure. Since M&E ation agenda can be enhanced with strong legal activities are largely dependent on the quality and support (CONEVAL), a permanent budget line, availability of internal information produced by or by using the agency’s own �nancial resources the programs and on their willingness and capaci- to carry out the M&E activities (DIPRES). In con- ty to generate primary data or recover information trast, when resources for M&E activities come out on bene�ciaries, the agency has many incentives of the program’s budget, have to be earmarked in to produce and use the best input information, loans, and in general, resources depend highly on and internally use the results. buy-in and voluntary adoption, then the agency Some disadvantages of the completely de- could end up neglecting precisely the programs centralized arrangement are less standardization and sectors most in need of monitoring and in methodologies and practices leading to more evaluation. heterogeneous quality and rigor in M&E products; External contracting. When M&E systems are limited coordination across sectors; fewer oppor- contracted out to consultants, �rms, or research centers and use competitive, open procurement pure research, and although the generation and processes, it increases legitimacy and indepen- use of research has been extensively and formally dence and reduces potential conflict of interest studied, the essence and rationale behind the two between evaluators and managers. DIPRES in are very different. Chile and SINERGIA in Colombia are good Capacity building and standardization. Ca- examples of systems using external contracting. pacity building and standardization are always Hybrid governance. A good example of a important, but become even more so under highly independent M&E body that still belongs highly decentralized settings, when a consider- to the government is the case of CONEVAL. Al- able volume of M&E activities is reached or has though it belongs to the executive branch through been mandated, and when agencies are devoting the social sector, it enjoys technical and manage- signi�cant �nancial and human resources to M&E rial autonomy greatly supported by a particular (CONEVAL). governance structure: an executive board of six independent academics, the Minister of Social Development, and its own executive director. Quality-control mechanisms. It is believed that the higher the buy-in of M&E by the agencies, M&E information can serve multiple purposes: the better the insight, quality, and completeness feeding back into policy and budget decision of information provided by the programs for making and national planning; improving policy M&E will be. However, in centralized settings, analysis and policy development; helping in mana- the M&E body can seek higher quality and more gerial activities such as program management reliable input information from the executing and staff or institutional management; enhanc- agencies through the establishment of control ing transparency and accountability; and many mechanisms, such as external veri�cation, audits, others (Mackay 2007, 9). The usual yardstick or alliances with internal control offices as well as of success for the M&E system is the degree of with national auditing and statistic offices. utilization of the information produced; almost Public disclosure. The principle of openness all existing M&E bodies highlight the importance refers to making the evaluation process as trans- of utilization. Utilization can also be thought of parent as possible and the results widely available. as one major determinant of sustainability. The Far-reaching laws regarding public information Development Assistance Committee (DAC) prin- access have been recently introduced in various ciples for evaluations state: “for evaluations to be countries and indirectly support the M&E system useful, they must be used� (OECD-DAC 1991). disclosure ability (such as in Mexico and Chile). Utilization is a prerequisite for �nancial sustain- The risk of lack of autonomy to disclose M&E ability, why fund costly M&E tools or evaluations information when the system is located under if they are not used? an executive authority can be mitigated with De�ning measures of successful performance other provisions such as a long-standing tradition is not easy.3 Two dimensions have been commonly to public disclosure. SINERGIA’s evaluations explored (table 2): the �rst dimension, coverage, exemplify limited public disclosure, as well as is a measure of the extent of the evaluation prac- the absence of a broader public information law. tice in relation to a reference value or universe, Academic rigor. Evaluations conducted exter- usually the proportion of the budget evaluated, nally by researchers and academia affiliates often that is, the value of the programs that have been undergo additional quality �lters established by evaluated to the total budget amount, or the the academic community. They can also easily number of programs evaluated in relation to a support capacity-building activities. Peer review- multiyear agenda, or the number or programs in ing, screening in seminars, and the process of a programmatic classi�cation of the budget. The publication all contribute greatly to ensure rigor second dimension is consensus on the need to and quality in �ndings (for example, SINERGIA follow-up on recommendations, commitments, and CONEVAL). These important advantages and action plans derived from M&E activities. may come at the cost of immediate usefulness This can include simple measures from the num- if dissemination in accessible language is ne- ber of alerts derived from monitoring, number glected. In the extreme, evaluation blends into of recommendations from evaluations that are actually adopted, to more demanding measures evaluations that examine the extent of M&E utili- such as the proportion of the recommendations zation and measure their cost-effectiveness. Such implemented over the total number of recom- reviews could be conducted on a more systematic mendations formulated. basis to help measure an M&E system’s overall Finally, in terms of �nal goals of the systems, cost-effectiveness, although cost and the typical namely improving quality and efficiency of public delays between conducting an evaluation and mea- expenditure, no indicators have been clearly es- suring its impact would make this problematical. tablished to measure these outcomes or attribute Measures on other dimensions, such as these to the effect of the M&E systems. There are, transparency and perception of accountability however, a small number of reviews of individual by citizens resulting from well-performing M&E systems, have not been explored. If the system the desk review evaluations (evaluaciones de consis- is also intended to influence budget allocations, tencia). Aspects to improve are classi�ed according further utilization measures could include the to involvement of different parties: some aspects change in allocations as a result of utilization by are under direct control of the program, while budget and congress, or more indirectly, correla- others involve attention of various units within tion measures with changes in resource allocation. an agency, various agencies, or require attention There is a mechanism in the Republic of Korea of different government levels. In addition, the whereby the budget allocations of programs that sector agencies themselves classify those aspects as fail to reach their performance targets by a speci�c high, medium, or low priority according to their margin are automatically cut. Some attempts on contribution to the program’s goal. For the 2008 using this potential measure examine the correla- budget, 101 programs with rapid evaluations tion between evaluation results and the growth were included in the tracking system, with 930 in the budget of evaluated programs (Kim and aspects to improve. Out of these, 70 percent were Park 2007, 2008). Finally, for the most rigorous aspects considered to be under direct control of evaluations, usage could also be measured in terms the program (CONEVAL 2008a, 2008b, 2010). of the standards for scienti�c production (that is, SINERGIA’s utilization focus has been on the citations and publications). operations and management of programs, since One of the strengths of the Chilean system is evaluations are typically conducted with volun- that it maintains very speci�c information regard- tarily involvement of sector ministries/agencies, ing program changes and it monitors the extent in contrast to the Chilean model. President Uribe of recommendations derived from evaluations. was a powerful champion of M&E information in Since the standardized terms of reference provide Colombia. He demanded intensive use of goals for the production of speci�c recommendations, and indicators, especially during his �rst adminis- these recommendations serve as a basis for estab- tration (OECD-DAC 2008). The downside is the lishing institutional commitments that are later limited use from budget authorities and Congress. closely monitored by DIPRES. Recently, the system has given attention also to The 2008 report by DIPRES is a good example tracking commitments from recommendations of the use of M&E information: between 2000 derived from each evaluation. and 2008, 174 programs were evaluated under In sum, M&E system utilization assessments the governmental program for evaluations and have so far included proxies of coverage, client sat- impact evaluations. There is a classi�cation of isfaction surveys (the World Bank’s Independent changes derived from recommendations with �ve Evaluation Group carries out annual surveys of categories presented in �gure 1. Between 1999 key audiences within and outside the Bank), evi- and 2007 there were more than 3,500 commit- dence on adoption of recommendations and com- ments derived from recommendations, around mitments (CONEVAL and DIPRES), and some 500 annually in the early years and less since anecdotal evidence (SINERGIA). Although more 2006. Out of these, 82 percent were ful�lled, 11 work remains to be done, M&E systems utilization percent were partially ful�lled, and 6 percent have is beginning to be addressed more systematically. not been ful�lled. It is generally accepted that DIPRES M&E information is highly utilized in budget analysis and decision making, in imposing program’s ad- M&E capacity building deals with the develop- justments, and for reporting to Congress and civil ment of successful M&E systems: systems that society. However, managerial usage or ownership are sufficiently independent to be externally from the head of programs has been limited, given credible and socially legitimate; systems that the centrally driven nature of the system and the influence policy making and are sustainable over perceived absence of incentives for the agencies to time; and systems that are responsive to the needs engage in their own evaluations (Mackay 2007, 29). of clients and main stakeholders. As more coun- In Mexico, CONEVAL issued in late 2008 the tries are thinking of establishing M&E systems, policy Mechanism for the Follow-Up of Aspects Sub- lessons from other countries’ experiences can be ject to Improvement to establish a general procedure extremely useful. Speci�c circumstances shape for tracking aspects of improvement derived from the evolution and focus of each system, and no recipe should be exported in a naïve manner to Program (WSP). Before joining WSP in 2009, she other countries. However, an explicit strategy to served for three years as director of Colombia’s support the development of new M&E systems M&E system, SINERGIA. Previously, she worked should start by identifying the country prefer- in the Office of Evaluation of the Inter-American ences; clearly de�ning the clients and the main Development Bank, participating in project and role for the M&E body; and determining how to country program evaluations. She has also con- track performance of the M&E system. sulted for the International Initiative for Impact Evaluation (3ie). She holds a master’s degree in international development from Harvard Ken- This note draws extensively upon previous work nedy School. conducted for the International Initiative for Impact Evaluation (3ie) and the Department for International Development of the United King- 1. In the Colombian institutional context, social control dom (Briceño and Gaarder 2009). The author refers to the use of constitutional participatory mecha- acknowledges their generous support. nisms that enable control of the state; trust and efficacy of the entities designed to control the state—Congress, Fiscalia, or the media; and the accountability mecha- nisms of those elected in office (Sudarsky 2008). Bertha Briceño works as senior evaluation spe- 2. Some examples of principles are the OECD-DAC cialist at the World Bank’s Water and Sanitation Quality Standards (2010), the principles of the Agencia de Evaluación from Spain (Ministerio de la Presidencia de España 2010), and the principles of the World para el año 2009.� http://www.dipres.cl/572/ Bank–IEG (2010). articles-41339_doc_pdf.pdf. 3. The World Bank and the Centro Latinoamericano Kim, John, and Nowook Park. 2007. “Performance Bud- de Administracion para el Desarrollo (CLAD) have geting in Korea.� OECD Journal on Budgeting (7) 4. contributed to the assessment of the systems by actively Mackay, Keith. 2007. How to Build M&E Systems to promoting diagnoses and reviews of systems’ perfor- Support Better Government. Washington, DC: World mances in 12 countries (CLAD and World Bank 2008). Bank, IEG. Ministerio de la Presidencia de España. 2010. Agencia de Evaluación y Calidad. Principios Rectores y Código Ético. http://www.aeval.es/es/la_agencia/ Briceño, Bertha, and Marie Gaarder. 2009. Institutional- presentacion/principios/. izing Evaluation: A Review of International Experi- OECD-DAC (Organisation for Economic Co-operation ence. International Initiative for Impact Evaluation and Development–Development Assistance Com- (3ie) and the U.K. Department for International mittee). 1991. “Principles for the Evaluation of Development (DFID). Development Assistance.� Paris. http://www.oecd. CLAD (Centro Latinoamericano de Administracion org/dataoecd/31/12/2755284.pdf. para el Desarrollo), and World Bank. 2008. ———. 2008. Emerging Good Practice in Managing for Fortalecimiento de los sistemas de monitoreo y Development Results (MfDR) Sourcebook, Third Edi- evaluación (M&E) en América Latina y el Caribe, tion. http://www.mfdr.org/sourcebook/. a través del aprendizaje Sur-Sur y del intercambio ———. 2010. Quality Standards for Development de conocimientos. http://www.clad.org/investiga- Evaluation. Paris. http://www.oecd.org/datao- ciones/investigaciones-concluidas/fortalecimiento- ecd/55/0/44798177.pdf. de-los-sistemas-de-monitoreo-y. Park, Nowook. 2008. “Does More Information Improve CONEVAL. 2008a. “Informe de seguimiento a los Budget Allocation? Evidence and Lessons from aspectos susceptibles de mejora de programas fede- Performance-Oriented Budgeting in Korea.� Work- rales 2008.� http://medusa.coneval.gob.mx/cmsco- ing Paper presented at the Congress of International neval/rw/resource/coneval/eval_mon/2733.pdf. Institute of Public Finance, August, Maastricht, ———. 2008b. “Mecanismos para el seguimiento a los as- Netherlands. pectos susceptibles de mejora derivados de informes Sudarsky, John. 2008. “La evolución del capital social en y evaluaciones a los programas federales de la admin- Colombia, 1997–2005.� Bogotá, revista 747 (144). istración pública federal.� http://medusa.coneval. World Bank, IEG (Independent Evaluation Group). gob.mx/cmsconeval/rw/resource/coneval/EVALU- 2004. Influential Evaluations: Evaluations That ACIONES/Mecanismo_seguimiento_291008. Improved Performance and Impacts of Develop- pdf?view=true. ment Programs. Washington, DC. ———. 2010. “Mecanismos para el seguimiento a los ———. 2010. Evaluation Methodology Overview. aspectos susceptibles de mejora derivados de in- http://web.worldbank.org/external/default/main formes y evaluaciones a los programas federales de ?theSitePK=1324361&piPK=64252979&pageP la administración pública federal.� www.coneval. K=64253958&menuPK=5039271&contentM gob.mx/contenido/normateca/2339.pdf. DK=20790052. DIPRES. 2008. “Informe de Finanzas Públicas: World Bank Web site Proyecto de Ley de Presupuestos del Sector Público www.worldbank.org/poverty/nutsandbolts This note series is intended to summarize good practices and key policy �ndings on PREM-related topics. The views expressed in the notes are those of the authors and do not necessarily reflect those of the World Bank. PREMnotes are widely distributed to Bank staff and are also available on the PREM Web site (http://www. worldbank.org/prem). If you are interested in writing a PREMnote, email your idea to Madjiguene Seck at mseck@worldbank.org. For additional copies of this PREMnote please contact the PREM Advisory Service at x87736.