66033 EAST ASIA AND PACIFIC REGION Social Development Reports Gender Dimensions of Community-Driven Development Operations A Toolkit for Practitioners GAP funded gender equality as smart economics A World Bank Group Gender Action Plan Gender Dimensions of Community-Driven Development Operations A Toolkit for Practitioners © 2011 The International Bank for Reconstruction and Development / THE WORLD BANK 1818 H Street, NW Washington, DC 20433, U.S.A. Telephone: 202-473-1000 Internet: www.worldbank.org/eapenvironment/sea-asia E-mail: feedback@worldbank.org All rights reserved. November 2011 This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The findings, interpretations, and conclusions expressed in this paper do not neces- sarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on maps in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or accep- tance of denoted boundaries. R I GHTS A N D PE R MI SSI ON S The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemina- tion of its work and will normally promptly grant permission to reproduce portions of the work. For permission to photocopy or reprint any part of this work, please send a request with complete infor- mation to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, tele- phone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. Photo credits: Cover photos by the World Bank. iii Table of Contents Acronyms and Abbreviations . . . . . . v Background and Acknowledgments . . . . . . vii Executive Summary . . . . . . ix 1 InTroduCTIon . . . . . . 1 2 Some uSeful m&e ConCepTS for InCludIng gender In Cdd . . . . . . 5 The Role of M&E in CDD . . . . . . 5 The objectives of M&E in CDD projects . . . . . . 5 The results chain . . . . . . 6 A Generic Results Framework for CDD . . . . . .6 Gender in the Generic Results Framework for CDD . . . . . . 8 Defining Indicators . . . . . . 9 An Overview of Monitoring and Evaluation Techniques . . . . . . 9 Monitoring techniques . . . . . . 9 Impact evaluation techniques . . . . . . 11 Capacity building for M&E . . . . . . 12 Collecting and using the information . . . . . . 13 3 exampleS of gender IndICaTorS for Cdd programS . . . . . . 15 Program Processes . . . . . . 15 Process indicators from program reporting/management information systems . . . . . . 15 Indicators from individual interviews . . . . . . 17 Qualitative evidence—typically from focus groups . . . . . . 17 Program Outputs and Outcomes . . . . . . 17 Overview . . . . . . 17 Access to services—the example of education . . . . . . 18 Access to services—generalizing the approach to other service sectors . . . . . . 19 Work and Income . . . . . . 20 Time saving services. . . . . . 21 Increased agency or empowerment . . . . . . 22 Summary of example indicators . . . . . . 27 iv G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s annexeS . . . . . . 29 Annex 1. Evaluating the NSP in Afghanistan . . . . . . 29 Annex 2. Asking about perceived impact on paid work . . . . . . 30 References . . . . . . 31 lIST of fIgureS Figure 1. An example of a results chain for a CDD project . . . . . . 6 Figure 2. CDD programs in a logframe structure . . . . . . 7 Figure 3. CDD impact evaluation areas . . . . . . 11 Figure 4. Which method to use to generate indicator data . . . . . . 23 lIST of TableS Table 1. Generic logframe for CDD operations: where to focus M&E . . . . . . 8 Table 2. Extract from the PNPM Indonesia results framework . . . . . . 8 Table 3. Asking how much influence your views had on the projects selected for funding . . . . . . 17 Table 4. Asking for estimates of project-specific impact on time spent . . . . . . 21 Table 5. Asking for estimates of project-specific impact on election of female officials . . . . . . 25 lIST of boxeS Box 1. The Power of Measuring Results . . . . . . 5 Box 2. Participatory monitoring in the Indonesia Kecamatan Development Program (KDP) . . . . . . 10 Box 3. How gender indicators on program process have changed practice . . . . . . 16 Box 4. Evidence from gender indicators on project employment . . . . . . 17 Box 5. Measuring economic impact, pilot test evidence . . . . . . 21 Box 6. Using focus group findings to explain changes in indicator values . . . . . . 22 Box 7. Identifying male attitudes to female empowerment: evidence from the Philippines . . . . . . 25 Box 8. Choosing key empowerment indicators . . . . . . 26 v acronyms and abbreviations BPD Badan Pemwakilan Desa (Democratically elected village representative council) CAS/CPS Country Assistance/Partnership Strategy CDD Community-driven development CDF Community Development Fund EAP East Asia and Pacific FAO Food and Agriculture Organization of the United Nations FGD Focus group discussion IFAD International Fund for Agricultural Development GAP Gender action plan GPI Gender parity index KDP Kecamatan Development Program (Indonesia) LAO PDR LAO Peoples Democratic Republic LKMD Lembaga Ketahanan Masyarakat Desa (Village Community Resilience Board) M&E Monitoring and evaluation MIS Management information system NGO Nongovernmental organization NSP National Solidarity Programme (Afghanistan) OECD Organisation for Economic Co-operation and Development OECD-DAC Organisation for Economic Co-operation and Development Development Assistance Committee O&M Operation and maintenance PAD Project appraisal document (World Bank) PCR Primary completion rate PDO Project development objective PNPM Program Nasional Pemberdayaan Masyarakat PRF Poverty Reduction Fund RF Results framework SHG Self-help groups SMART Specific, measurable, achievable, relevant, and time-bound TQQ Time, quality, and quantity TTL Task team Leader UNIFEM United Nations Development Fund for Women vi G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s vii background and acknowledgments The World Bank recognizes that community-driven In this context, the EAP Sustainable Development Depart- development (CDD) approaches and actions are impor- ment has developed this toolkit to promote the wider use tant elements of an effective poverty reduction and sus- of gender indicators in the region’s CDD projects. It pro- tainable development strategy, which is why CDD is vides practical guidance to World Bank EAP operational becoming an increasingly important approach in the task teams and other CDD practitioners (i.e. government/ fight against poverty. In 2006, CDD projects comprised NGO staff) on how to measure the gendered impact of nearly 15 percent of the World Bank’s lending portfo- CDD operations. The toolkit is a result of a regional pilot lio in the East Asia and Pacific (EAP) region, including M&E initiative and is complemented by two additional forty-two active projects in eight countries that were publications from the EAP Sustainable Development either classified or had a CDD component. Department that address gender monitoring within spe- cific CDD operations in Lao PDR and the Philippines.2 Evaluations and research findings indicate that CDD projects—by working to give control over planning The toolkit improves our understanding of why gender decisions and investment resources for local develop- matters to the monitoring and evaluation of CDD proj- ment projects to community groups—can be excellent ects. It introduces M&E topics that the non-specialist vehicles for empowering women and promoting gender can find useful when constructing gender indicators. equality. Findings also indicate that women and men Furthermore, the toolkit presents a generic CDD results can often have different priorities for CDD, and that framework that provides convenient categories for incor- the involvement of females in decision-making about porating gender M&E indicators, tangible examples of public services can improve service delivery at the com- gender indicators, and illustrates how gender M&E can munity level.1 be added to CDD program results frameworks. Yet, in the Gender in Agriculture Sourcebook (2009,) This toolkit was prepared in close consultation with com- the World Bank, IFAD, and FAO point to evidence munity-driven development practitioners, as well as gen- that untargeted CDD projects often bypass women. der and social development experts within the World Bank. They also conclude that to date the documentation and evaluation of CDD on building accountability to The Bank’s Task Team included Nina Bhatt (Task Team rural women and transforming gender relations are Leader) and Helle Buchhave. Gil Yaron was the main extremely limited, and that increased attention to gen- author. The team received technical comments and/ der in the M&E of CDD projects is especially critical to or guidance from Sean Bradley, Markus Kostner, Anne ensure that CDD projects have the intended impact for Kuriakose, Julien Labonne, Ian Parker, Helene Carls- inclusive poverty reduction. son Rex, and Susan Wong. Moreover, the report ben- efitted from the guidance of the peer-reviewers, which 1 See, for example: (a) McLaughlin, Karrie, Adam Satu, and Michael included Kathleen G. Beegle, Nora Dudwick, and Jan- Hoppe. 2007. “Kecamatan Development Program Qualitative Impact mejay Singh. Specific inputs were provided by Bob Evaluation.� Jakarta: World Bank, Indonesia; (b) World Bank. 2008. Livernash (editing) and Nina Queen and Florian Kitt, Community-Driven Approaches in Lao PDR. Moving Beyond Service De- who coordinated the report production. The report was livery. Human Development Sector Unit EAP. Washington DC: World made possible with the financial support of the World Bank; (c) World Bank, FAO, and IFAD. 2009. Gender in Agriculture Sourcebook. Washington, DC: World Bank; and (d) Chattopadhyay, R., Bank Gender Action Plan. and E. Duflos. 2004. “Women as Policy Makers: Evidence from a Ran- domized Policy Experiment in India.� Econometrica 72 (5): 1409–1443. 2 Find them here: www.worldbank.org/eapsocial There is evidence that untargeted community-driven development projects can bypass women. Gender indicators within the program results framework can enable practitioners to identify better ways of delivering their poverty reduction objectives, yet gender indicators are not widely used. Photo: © Curt Carnemark / World Bank ix executive Summary C ommunity-driven development (CDD) pro- grams require monitoring and evaluation (M&E) to tell those implementing and funding the programs whether they are on track to deliver, or have delivered, desired outcomes such as improved ser- dant evidence that untargeted CDD can bypass women and the poor . . . Women’s marginalized status within the community renders their voices less significant than those of men; they have less access to decision making and to the resources for development, and limited time vices, economic activity, and empowerment. Monitor- and mobility to attend meetings that determine wom- ing information—including an early warning system to en’s needs and priorities.� It is also relevant to note that identify problems that can still be put right—is essential men and boys can also be negatively affected by the fail- for project managers to manage effectively. Impact eval- ure to consider gender. A World Bank (2011b) pilot of uation measures results and enables us to learn what CDD gender indicators in the Philippines highlights the has worked and why. M&E thus plays a crucial role in increasing problem of higher school dropout rates for enabling programs to deliver poverty reduction. boys than girls; in 2006, only 69 percent of boys com- pared to 78 percent of girls persisted to the last year of In CDD programs, local communities decide which primary school. projects are priorities, whether the broad areas of focus are improved services, economic activity, or The objective of this toolkit is to provide practical empowerment. As communities reflect the interests guidance to World Bank EAP operational task teams of their constituent groups, monitoring and evalua- and other CDD practitioners (i.e. government/NGO tion of CDD programs must take socioeconomic and staff ) on how to measure the gendered impact of gender differences into account in order for programs CDD operations. First, this is necessary because CDD to effectively support poverty reduction across the program reviews have found that gender indicators whole community. are not widely used. Second, several governments in the EAP Region have identified gender as an impor- Put slightly differently, if the program objective is to tant pillar in poverty alleviation strategies, in the reduce poverty, it is important to know that CDD proj- light of evidence suggesting that societies promoting ects reflect the needs of the poor. If local elites decide more equal opportunities for men and women have on project priorities, the program will fail to meet this higher growth, lower poverty, and better develop- objective. Likewise, poor men and women can have dif- ment outcomes. Third, gender mainstreaming is a ferent priorities; CDD projects have to reflect the needs critical facet of World Bank policy and programs. of both. Ignoring the priorities of up to half the poor Fourth, as this toolkit demonstrates, it is straightfor- will seriously weaken the program. In the same way, it ward to add gender indicators to a results framework. is exceptionally important to identify whether and how It involves disaggregating some of the indicators that both men and women gain from the program. If up to will already be in the results framework by gender, as half the target group is missing out, the program is fail- well as adding a limited number of specific gender ing to deliver. indicators. This is not merely a theoretical concern. Based on an This toolkit takes CDD practitioners and other inter- extensive review of CDD projects, the World Bank, ested readers through the necessary steps to identify FAO, and IFAD (2009) conclude that “. . . there is abun- where to track gender in the results framework, as well x G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s as suggesting possible indicators. This toolkit is orga- In contrast, program output and outcome indicators nized in three sections: are more sector-specific. The intention is to illustrate the types of indicators that can be used. In some cases, 77 Section 1 sets out why gender matters for CDD program teams will be able to use example indicators M&E. directly in their results frameworks; in others, indica- 77 Section 2 provides an introduction (and pointers to tors will need to be modified for the specific sector as further reading) on M&E topics that the nonspe- well as local context. cialist will find useful when constructing gender indicators. This includes a generic CDD results The table below sets out examples of gender indicators framework structure that provides convenient cat- by type of indicator (process, output, and outcome) and egories for incorporating gender M&E indicators. sector. Indicators in the “additional indicators� section 77 Section 3 uses these categories to provide may be very important for some programs and are given examples of indicators (and other evidence) this title simply because other indicators can provide from the EAP region and illustrates how gen- related information. der M&E can be added to CDD program results frameworks. Qualitative research (often associated with focus group discussions) plays an important role in measuring the CDD programs tend to face similar issues of whether gendered impact of CDD operations. Rather than pro- priority projects are, in fact, priorities for both men ducing indicators, its value lies in providing the essen- and women. This applies if the sector focus is improved tial context required to interpret numerical indicators: services, economic activities, or empowerment. Hence to explain why values are high or low and how changes process indicators are likely to be common across sec- have occurred. The combination of quantitative indica- tors (although the wording will need to reflect local tors and qualitative, contextual research will produce institutional and political context). the most reliable findings. G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s xi Gender indicator examples (as numbered in the document text) Process indicators 1. Percent of village development committee members who are women 2. Are there single-sex groups in the planning process that identify priority projects? 3. If so, what proportion of projects implemented were proposed by (a) women-only groups, (b) by men-only groups, or (c) were priorities for both groups? Output & Outcome indicators—Improved services (education) 10. Percent change in female enrollment in primary schools 11. Percent change in male enrollment in primary schools 12. Percent change in female enrollment in secondary schools 13. Percent change in male enrollment in secondary schools 14. Percent change in female completion of primary school 15. Percent change in male completion of primary school Output & Outcome indicators—Improved services (health & water & sanitation) 16. Percent change in access to health services for men 17. Percent change in access to health services for women 18. Percent change in access to water and sanitation facilities for men 19. Percent change in access to water and sanitation facilities for women 20. Percent increase of girls and women receiving health benefits (full immunization, anti-natal, and prenatal and postnatal care, maternal health care) Output & Outcome indicators—Increased incomes 21. The percent change in working-age women engaged in paid work as a result of the project 22. The percent change in working-age men engaged in paid work as a result of the project Outcome indicators—Improved services (time-saving) 23. Percent of women reporting a reduction of time spent on daily household tasks as a result of the program 24. The proportion of women who use the new asset or service 25. Average time saved per day for each woman using the new asset or service Outcome indicators (empowerment) 26. The percent change in local female elected officials (village and municipal offices) as a result of the project 27. The percentage of women who report having no power to make decisions regarding (a) what to buy at the market; (b) asset purchases; (c) number of children; (d) schooling of children; and (e) use of family planning 28. Percentage of women who know how much income and expenditure there is in the household 29. Percentage of women who can travel outside the village to visit relatives without permission 30. Frequency of domestic violence experienced by women in the past year* additional gender indicators Process indicators 4. Number of men gaining voluntary employment on program activities in the past year and over the life of the program 5. Number of women gaining voluntary employment on program activities in the past year and over the life of the program 6. Number of men gaining paid employment on program activities in the past year and over the life of the program 7. Number of women gaining paid employment on program activities in the past year and over the life of the program 8. Proportion of men who believe their views influence projects selected at the village level 9. Proportion of women who believe their views influence projects selected at the village level *Based on four categories: (a) Zero, (b) 1-2/year, (c) 3-5/year, and (d) 6+/year. The process by which a community decides what to invest in will influence who benefits and how. Photo: Indonesia. © Ray Witlin / World Bank 1 Introduction 1 C ommunity-driven development (CDD) is an approach to poverty reduction that gives con- trol of decisions and resources to community groups. As noted by Dongier et al. (2002): other CDD practitioners (i.e. government/NGO staff ) on how to measure the gendered impact of CDD opera- tions. There are a number of powerful reasons why this toolkit focuses on gender aspects of CDD monitoring and evaluation. “These groups often work in partnership with demand- responsive support organizations and service providers, First, CDD programs need to ensure the development including elected local governments, the private sector, needs of both men and women are met in order to effi- NGOs, and central government agencies. CDD is a way ciently and effectively deliver poverty reduction; that is, to provide social and infrastructure services, organize gender matters in the context of CDD. Evidence to sup- economic activity and resource management, empower port this can be found at all stages of the CDD project poor people, improve governance, and enhance security cycle: of the poorest.� 77 Women and men can often have different priori- As with other interventions to reduce poverty, CDD ties for CDD. Chattopadhyay and Duflo (2001) programs and projects require monitoring and evalua- compared decisions made in villages with and tion (M&E). This tells those implementing and funding without women’s representation in village-level the programs whether they are on track to deliver or councils (panchayats). In the two Indian states have delivered the desired outcomes of improved ser- included in the study, panchayats with women vices, economic activity, and empowerment. members invested more in goods that were rel- evant to the needs of local women. Yet CDD monitoring and evaluation needs to do more 77 Where priorities differ, the process of deciding than simply look at program and project outcomes. which needs to prioritize for investment directly The process by which the community decides where to affects who ultimately benefits. This process invest will influence who benefits and how.3 As a con- should be monitored and corrective action taken sequence, M&E should involve looking at project pro- where investments are skewed toward either cesses as well as outcomes that result from projects. men’s or women’s interests.4 Failure to identify Moreover, as communities reflect the interests of their and correct this bias can seriously reduce the constituent groups, M&E must take socioeconomic and ability of a project to reach the poor. Reviews of gender differences into account in order for programs to IFAD CDD projects (IFAD 2004, 2006) underline effectively support poverty reduction across the whole these concerns and led the World Bank, FAO, community. and IFAD (2009) to conclude that “. . . there is abundant evidence that untargeted CDD can The objective of this toolkit is to provide practical guid- ance to World Bank EAP operational task teams and 4 Experienced project staff may find that provisions that worked in one area at a given time fail to curb excessive traditional authority influ- 3 For example, where customary traditions deny rights and privileges ence in another. By monitoring the process used to select CDD proj- to women, relying on customary community institutions for project ects at each phase in the program, the Lao PDR PRF identified that the implementation can deepen gender inequality (Beall 2005, cited in original process for project selection was disadvantaging women and World Bank, FAO & IFAD 2009). introduced new processes to address this (World Bank 2011a). 2 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s “Sometimes men look down on us women. Sometimes they don’t accept our point of view. That discourages us.� Woman from Solomon Islands. Photo © thinkEQUAL / World Bank bypass women and the poor . . . Women’s mar- Gender indicators within the program results (M&E) ginalized status within the community renders framework therefore enable practitioners to iden- their voices less significant than those of men; tify better ways of delivering their poverty reduc- they have less access to decision making and to tion objectives. Yet gender indicators are not widely the resources for development, and limited time used. The evidence from a wide-ranging review of CDD and mobility to attend meetings that determine and Community Development Fund (CDF) projects by women’s needs and priorities.� the World Bank, FAO, and IFAD (2009) suggests that 77 CDD project outputs, outcomes, and impacts the experience of IFAD is typical in that “current infor- clearly determine who within the community mation on gender aspects and impacts in the CDFs is benefits. Gender indicators provide both criti- superficial; assessments of CDD and CDFs have not cal information on delivery of poverty reduc- measured gender impacts or participation of women in tion and, if used skillfully, an opportunity for the capacity-building activities.� course correction. It is important to note that men and boys can also be negatively affected by Second, several governments in the EAP Region have the failure to consider gender. A World Bank identified gender as an important pillar in poverty (2011b) pilot of CDD gender indicators in the alleviation strategies in the light of evidence suggest- Philippines highlights the increasing problem of ing that societies promoting more equal opportunities higher school dropout rates for boys than girls; for men and women have higher growth, lower pov- in 2006, only 69 percent of boys compared to 78 erty, and better development outcomes (World Bank percent of girls persisted to the last year of pri- 2010b). If you cannot measure the gender outcomes mary school. of a CDD program, you are losing the opportunity G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 3 to demonstrate how your work fits into this bigger intended to enable CDD practitioners to construct a picture. similar indicator for a new sector. Third, gender mainstreaming is a critical facet of It is also worth noting that most of the indicators in this World Bank policy and programs. The EAP Region toolkit have been piloted within the EAP Region and has a Regional Gender Action Plan, endorsed by senior draw on the valuable experience gained from piloting management, which has the development of country gender indicators in CDD programs in Lao PDR, the gender action plans at its core (World Bank 2010b). Philippines, and in Indonesia. These will cover the same period covered by the Coun- try Assistance/Partnership Strategy (CAS/CPS) and will This toolkit is based on three steps: include impact, outcome, and output gender indicators. Finally, as this toolkit demonstrates, it is straightfor- 1 2 3 Introducing why Useful M&E Examples ward to add gender indicators to a results frame- gender matters concepts & of gender work. It involves disaggregating by gender some of the for CDD M&E techniques indicators output and outcome indicators that will already be in a results framework, as well as adding a limited number of specific gender indicators. This toolkit describes the The remainder of this document elaborates steps 2 and necessary steps to identify where you should be track- 3. Section 2 provides an introduction (and pointers to ing gender in the results framework, as well as suggest- further reading) on M&E topics that the nonspecialist ing possible indicators. It is important to note that not will find useful when constructing gender indicators. all the indicators in this toolkit will apply to a specific Having set out the reasons for using a logical framework program or project: if there are no CDD interventions approach, we consider a generic CDD results framework in the education sector, then there is no need to select that provides a convenient structure for incorporating the suggested education indicators. Conversely, a proj- gender M&E. Section 3 provides examples of indicators ect may have outputs outside the sectors that are men- (and other evidence) that will help practitioners add tioned. The examples given from a range of sectors are gender M&E to their CDD program results framework. In 2006, CDD projects comprised nearly 15 percent of the World Bank’s lending portfolio in the East Asia and Pacific (EAP) region, including forty-two active projects in eight countries that were either classified or had a CDD component. Photo: Laos. © Helle Buchhave / World Bank 5 Some useful m&e Concepts for Including Gender in CDD 2 The Role of M&E in CDD example above, it is evidence from project monitoring that allows previously unforeseen The objectives of M&E in CDD projects constraints to be addressed—to get the project Monitoring and evaluation (M&E) enables manag- back on track. ers to track progress and get better results by learning 77 Provides opportunities for beneficiaries to what works, what doesn’t, and why this is the case. The get feedback on progress and raise concerns if importance of doing this is summed up in Box 1 below. necessary. The role played by monitoring differs from that of evalu- ation. Specifically: Evaluation 77 Helps us to measure the impact of the project Monitoring or program and understand the major drivers of 77 Measures progress against work plans and this impact. Given the investment in the project, budgets. it is important to identify which groups have 77 Gives an early indication to project managers benefited and by how much. Capturing the gen- of what is working and what isn’t. It flags areas der dimension to this story is critical to under- where more detailed investigation is needed to standing the impact on poverty—an issue taken understand why this is the case. For example: up in the following section. Are you reaching your target beneficiaries as 77 Tells us whether the project approach has deliv- planned, and if not why not? This requires you ered or is likely to deliver the project goals. to know whether the project is systematically 77 Enables changes in the well-being of CDD proj- failing to reach poor women or poor men—an ect beneficiaries to be attributed to a particular issue we return to in Section 3. project or program. 77 Provides rapid feedback for decision mak- 77 Is required to test (pilot) innovative approaches ing and course correction if necessary. In the to poverty reduction before these can be repli- cated at a larger scale. bOx 1 The power of measuring results For Further InFormatIon • If you do not measure results, you cannot tell S. Wong. 2009. Monitoring & Evaluation For CDD Operations: success from failure. Lessons Learned. Washington, DC: World Bank. Available at: • If you cannot see success, you cannot reward it. http://www.adb.org/Documents/Events/2009/CDD-Rural- • If you cannot reward success, you are probably Poverty-Alleviation/S361-Susan-Wong.pdf. rewarding failure. • If you cannot see success, you cannot learn from it. J. Kusek and R. Rist. 2004. Ten Steps to a Results-Based • If you cannot recognize failure, you cannot correct it. Monitoring and Evaluation System: a handbook for • If you can demonstrate results, you can win public development practitioners. Washington, DC: World Bank. support. Available at: http://www.oecd.org/dataoecd/23/27/ Source: Osborne and Gaebler 1992, quoted in Kusek and Rist 2004. 35281194.pdf. 6 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s fIGure 1 an example of a results chain for a CDD project Outputs/Intermediate Outcomes Outcomes Increased access of communities to basic infrastructure, services, and Empowered communities income-generating activities Community control and management Favorable policy and legal of decisions and resources environment and strong local government institutions Impacts Inputs Sustained development and positive Resources for nancing capacity impact on lives of the poor building at the community level, for preparing and implementing development plans according to community priorities. Support for strengthening legal and policy environment and local government institutions Source: Adapted from World Bank 2005b. The results chain A Generic Results Each CDD project has a theory of how it will reduce Framework for CDD poverty.5 This can be represented as a results chain (see Different organizations have different approaches to figure 1 for an example) that sets out a logical sequence monitoring and evaluating progress along the results of how the project will turn inputs (resources allocated chain. The World Bank uses a results framework to project activities) into outputs and intermediate out- (RF)—one type of logical framework (logframe)—to comes (more assets and more responsive institutions) summarize what the project is trying to achieve and that will deliver outcomes by fulfilling project objec- to set out how results are measured, monitored, and tives, such as increased access to services and better evaluated. livelihood options for the poor. A successful project will then ultimately have an impact in terms of poverty Although each results framework is (or should be) reduction. tailored to a specific project or program, CDD inter- ventions do have common generic characteristics. By For Further InFormatIon focusing on a generic logical framework, it is possible to consider M&E in a way that should be meaningful W.K. Kellogg Foundation. 2004. W.K. Kellogg Foundation for a wide variety of CDD projects. Based on a review Logic Model Development Guide. of World Bank CDD projects, Jorgensen (2005) pro- posed a generic logical framework for CDD opera- 5 This “theory of change� is also very helpful in identifying areas to tions that was subsequently taken up by the World focus in the course of developing monitoring and evaluation plans; Bank (World Bank 2007a). This is represented in fig- see Leeuw and Vaessen (2009). ure 2 below. G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 7 fIGure 2 CDD programs in a logframe structure Final Outcome Well-being Program Objectives Income Empowerment Services More and Better Program Outputs/ Better Institutions Distributed Assets Intermediate (Inclusive, Cohesive, (Physical, Financial, Outcomes Accountable) Human,Natural, Social) Program Activities Institution-building Asset Investment Source: Adapted from Jorgensen (2005)/World Bank (2007a). While there is likely to be a common understanding achieved.6 As a practical example of how this works, the of income generation and service improvement as logframe suggests we look for impact of a CDD micro- project development objectives (PDOs) in CDD proj- finance project on female agency as a result of increas- ects, there are various definitions of empowerment ing assets (such as microfinance) and from the potential in current use. In this toolkit we highlight two sepa- influence of the project on institutions—such as sup- rate aspects of empowerment: agency and collective porting the establishment of self-help groups. action. This “unpacking� of empowerment is impor- tant for M&E and, in particular, for monitoring and With this in mind, table 1 considers likely areas of focus evaluation of gender in CDD. In the social sciences, for M&E within the hierarchy of generic CDD objec- agency refers to the capacity of individuals to act tives. It is based on a review of the World Bank CDD independently and to make their own free choices; it portfolio and reflects the three main project develop- can be exercised in the individual, household, or pub- ment objectives (PDO): (1) income generation, (2) lic sphere (Barker 2005). Following Jorgensen (2005), empowerment, and (3) service improvement. Any par- collective action refers to the ability of people to work ticular CDD program or project is likely to focus on just collectively. one of these areas and is likely to cover just a subset of the program activities in table 1; for example, a proj- There are unresolved debates over the determinants ect to improve district health and education may not of agency and there is no unique model that explains change natural capital. female agency in development. The generic logframe simply suggests that for CDD projects, increased agency It is worth noting that there is almost always an institu- is likely to reflect increased access to assets (sometimes tion-building dimension to CDD projects even if asset described as endowments) and better institutions that creation or service delivery is the most visible interven- influence how these assets can be used. The logframe is a convenient way of presenting the logic of CDD inter- 6There are also likely to be interactions between program objectives; for example, the choices available to working women are likely to in- ventions. It suggests where to focus our M&E efforts, crease if they have to spend less time collecting water. As we will see, but does not describe the full story of how outcomes are qualitative research is required to understand these interactions. 8 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s tion. This reflects the fact that CDD investments are Table 1 Generic logframe for CDD operations: where to decided by local institutions and their performance is focus m&e likely to have a major impact on the subprojects selected Hierarchy of for investment. Objectives Possible areas of emphasis for M&E Final Outcome Improved well-being for target group In Table 1, program processes are treated as separate from activities or outputs. This helps to focus attention Program Sustained, higher and less volatile income on monitoring who decides on project activities and Objectives: Better services that directly affect well-being how marginalized target groups make their views heard. Desired outcomes (including time saving) These decisions are likely to have a profound impact on program outcomes and therefore it is very important to Increased empowerment (agency and collective action) identify process indicators. However, once this has been Program Individual/household/community control over more done, process indicators are typically placed alongside Outputs & and better-allocated assets (endowments) activity or outcome indicators in the results framework Intermediate • Physical, financial, human, natural, and social Outcomes: capital (RF). This can be seen in the extract from the PNPM RF Assets and • Better functioning and more equitable institutions (Table 2). institutions Program Institution-building (organisations, markets, legal Gender in the Generic Results Activities: rights & social norms) What is done? Framework for CDD Asset creation The areas within the logframe where gender is particu- • Investments in human, financial, physical, natural larly relevant are: and social capital Program Targeting Program processes. Process indicators tell you how the Processes • By geography/community characteristics program is implemented and who is involved. Monitor- How is the • By personal household characteristics ing and acting on gender process indicators is critical to program implemented? Who does what: ensure that CDD projects meet the needs of both men Who is • Identification, planning of project activities and women. involved? • Implementation • Channeling or management of funds Program Outputs and Outcomes. Program inter- • Monitoring and evaluation ventions have impacts on individual endowments or Source: Adapted from Jorgensen 2005. assets (such as education, health, land, and financial Table 2 extract from the pnpm Indonesia results framework Intermediate Results Results Indicators for Each Component Use of Results Monitoring Component One: Block Grants Component One: Component One: Villagers participate in a process Min. 40% participation rate of women and poorest Assess if planning and inclusion to plan, select, and manage community members in planning and decision-making procedures and policies need basic social and economic meetings adjustment to encourage greater infrastructure provided through 85% of agreed work plans completed each year participation block grants Assess if subdistrict sites are benefiting #/type of infrastructure works, economic, and education and health subprojects/ activities completed from KDP financing and assistance in 4,000 subdistricts by 2009 Determine if program needs to >70% of infrastructure works are evaluated as of high increase its inspection and supervision quality of technical works and O&M arrangements O&M arrangements are in place and functioning for >70% of infrastructure works Source: Annex 3: Results Framework, INDONESIA: PNPM, PAD 2008 prepared by S. Wong G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 9 resources) and institutions that determine the oppor- For Further InFormatIon tunity individuals have to use their endowments (such as social norms, civil society, markets, and public sec- F. Leeuw and J. Vaessen. 2009. Impact Evaluations and tor institutions). Gender indicators, just like other pro- Development: Nonie Guidance on Impact Evaluation. gram indicators, may reflect program outputs related Washington, DC: World Bank, Independent Evaluation to assets and institutions and the outcomes that result. Group. Available at: http://www.worldbank.org/ieg/nonie/ These program outcomes will be various aspects of guidance.html. well-being, such as higher incomes, better access to ser- vices, or increased agency (an aspect of empowerment). Two types of gender indicators are required to monitor An Overview of Monitoring and and evaluate project interventions in these areas. First, Evaluation Techniques there is the simple step of disaggregating key existing The aim of this subsection is to provide an overview of indicators by gender to capture how program invest- key M&E techniques and some pointers to further read- ment affects access to services, acquisition of assets, and ing for those who are interested. Things we identify as new sources of incomes or improved incomes. Second, good practice at each stage of CDD program M&E apply a limited number of new gender indicators are needed to gender M&E and CDD M&E in general. to capture impacts on time saving and empowerment (particularly the agency aspect of empowerment, such Monitoring Techniques as control over assets, income, and travel and the inci- Basic program monitoring requirements are to track: dence of gender-based violence). 77 Progress against the work plan (inputs and activities) Defining Indicators 77 Whether the budget is being used as planned. The distinction made by Hentschel (1999) between data and the methods used to collect it is relevant to how we Internal project monitoring information systems are define indicators in this toolkit. As stated by Garabino typically used to do this. Staff need to be trained to col- and Holland (2009), “Quantitative research produces lect and use valid, reliable, and timely information and data in the form of numbers, while qualitative research tends to produce data that are stated in prose or textual forms.� Numerical indicators are used in this toolkit in order to have values that can be reliably compared across project sites and over time. When it comes to setting targets for these indicators, they should be SMART; that is, spe- cific, measurable, achievable, relevant, and time-bound. A good way of ensuring your indicators are SMART is to make sure each one talks about time, quality, and quantity (TQQ); for example, “85 percent of agreed work plans completed each year.� Qualitative research (most often associated with focus group discussion) should not be used to produce indi- cators. Rather, its value lies in providing the essential context required to interpret numerical indicators: to explain why values are high or low and how changes Ethnic Hmong women in Ban Khan Khao Village, Hua Meuang District, Lao have occurred. The combination of quantitative indica- PDR, are being interviewed in pairs to provide support for each other and tors and qualitative, contextual research on these issues help with Lao language comprehension during an initiative to review gender from representative locations will produce the most indicators for the PRF CDD projects. Photo © Anders Engvall/World Bank reliable findings. 10 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s there can be a gender issue of who gets training.7 Capac- monitoring by the community themselves—encouraging ity building for M&E is discussed further below. local people to hold those delivering improved services or infrastructure to account. Participatory monitoring In addition, process indicators often play a valuable role is described for the Indonesia Kecamatan Development in routine monitoring, providing essential early warn- Program (KDP) in box 2 below. ing if target groups are being excluded. An example from the PNPM results framework (see Table 3 above) Important messages for monitoring are to: is the requirement to track whether there has been a: 1. Make sure data is valid, reliable and timely “Minimum 40 percent participation rate of women and (Kusak and Rist 2004). poorest community members in planning and decision- 2. Make sure data gets used to improve project making meetings.� delivery (Wong 2003). If monitoring against indicator targets suggests things are not going according to plan, you need to know why. For Further InFormatIon Qualitative investigation—often using case studies—is a powerful way of finding this out. The program should J. Kusak and R. Rist. 2004. Ten Steps to a Results-Based lead this process, but may need to bring in outside Monitoring and Evaluation System: a handbook for develop- expertise. ment practitioners. Washington, DC: World Bank. Available at: http://www.oecd.org/dataoecd/23/27/35281194.pdf. There are important aspects of CDD program monitoring that are external to the program, such as (a) independent S. Wong. 2003. “Indonesia Kecamatan Development Pro- monitoring by civil society groups and village commit- gram: Building a Monitoring and Evaluation System for a tees; and (b) formal grievance and complaint resolution Large-Scale Community-Driven Development Program.� mechanisms. Perhaps the most important is participatory EASES Discussion Paper. Washington, DC: World Bank. Avail- able at: http://siteresources.worldbank.org/INTINDONESIA/ 7 For example, if more female staff work part-time, it would be a mis- Resources/Social/KDPM&Epaper.pdf. take to only offer M&E training to full-time staff. bOx 2 participatory monitoring in the Indonesia Kecamatan Development program (KDp) Over the years, KDP encouraged different kinds of community participatory monitoring: • Monitoring by village councils (BPDs). Laws for the election of village councils were passed in 1999. Prior to this, village chiefs generally appointed the village assemblies (LMDs) and village community resilience boards (LK- MDs) from village elites. By the middle of KDP however, villagers were democratically electing their representa- tive councils in many Indonesian villages. BPDs have responsibility for monitoring KDP activities at all stages: socialization, planning, implementation, and maintenance. The BPDs select members to monitor each phase. Results of this monitoring are then discussed at council meetings or fed into larger village meetings. • Monitoring by special community groups or teams. KDP encouraged community monitoring groups in each vil- lage during years 2 and 3. Communities were encouraged to form special teams or groups at the community level during the village meetings to monitor KDP. These community monitoring groups were independent of the village implementation teams. The community team members shared responsibilities for checking financial ac- counts, monitoring bank transactions and material purchases or rentals, visiting suppliers to confirm the costs of goods, and monitoring subproject activities, including infrastructure construction. • Community participatory monitoring facilitated by NGOs. In several provinces, including Aceh and East Java, NGOs involved in province-based monitoring helped a number of villages, special groups, and teams conduct com- munity participatory monitoring. They helped villagers decide what questions were important to them about KDP, how to collect data to answer those questions, and helped villagers analyze community findings. NGO facilitation has been a success story in several locations, and the activity will be expanded to every province under KDP2. Source: Wong 2003. G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 11 Impact evaluation techniques fIGure 3 CDD impact evaluation areas Impact evaluation asks: “What has been the impact of our CDD program?�8 CDD program objective Evaluation question examples The generic CDD logical framework presented earlier in Figure 2 and Table 1 suggest the kinds of impacts we are likely to be looking for. This is brought together with • Has CDD reduced poverty? examples of evaluation questions in Figure 3 below. • Have female-headed households Higher income gained more or less from CDD than Most CDD programs will have one or possibly two of other groups? these objectives and so the impact evaluation will focus on these areas. • Has CDD increased access to education Impact evaluation is not just about whether project for boys & girls? objectives have been achieved, but also whether the • HAs CDD improved health outcomes for Better services interventions in a CDD that are stated to reduce pov- men and women? erty (as set out in a results framework) have been or • Have CDD projects in water & energy produced time-savings for women? are being realized. Hence there is a need to consider progress against program output and intermediate out- come indicators—particularly if the program has not been running long enough to achieve the full program • Has CDD increased the % of men & women who believe their views objective. As with outcome indicators, incorporating Empowerment in uence local government? a gender perspective involves disaggregating existing • Has CDD reduced the incidence of indicators by gender and adding a small number of new gender-based violence? gender-specific indicators. Examples of these indicators are given in the following section. Qualitative research is also important to understand why progress has or has not been made at each stage—from inputs to outputs/ 2. In order to make this comparison, baseline intermediate outcomes and through to impact. data must be collected at the start of the proj- ect. Baseline data also play a crucial role in At this point it is important to mention attribution; that monitoring progress over time, and reliable is, how you can be sure that the impact identified is actu- data need to be put in place at the start of the ally due to the CDD program. There are a number of program. techniques available and interested readers are referred 3. In programs where control sites are not avail- to Leeuw and Vaessen (2009) for detail. Here we note: able, the same indicators can be used to reliably identify impact if outside influences (ranging 1. Where control as well as project (intervention) from government policy to other project activi- sites are used, it is much easier to separate out ties) are unlikely to have influenced these indica- program impacts from broader changes in soci- tors. However, this is only plausible if there are ety or NGO activities. Program-specific impacts minimal project interventions from government can simply be identified by comparing changes in or NGOs on these issues in your project sites. indicator values for project and control sites over Where such interventions do occur, a second- the same period of time.9 This applies to gender as best means of identifying your program or proj- well as other outcome and impact indicators. ect-specific impact is to ask respondents directly how much difference the project has made to a 8 This means both intentional and unintentional impact. The Devel- particular indicator.10 opment Assistance Committee of the OECD defines impact as “the 4. As a general rule (which applies to interpret- positive and negative, primary and secondary long-term effects pro- duced by a development intervention, directly or indirectly, intended ing gender and other impact indicators), a or unintended� (OECD-DAC 2002). 9 Assuming control sites have been correctly identified. There are also 10The questions needed in this case are slightly different, as respon- ethical issues to consider when using this methodology—see Leeuw dents typically find it difficult to calculate precise changes (in time or and Vaessen (2009), Chapter 4. money); that is, the questions need to cover broad categories of impact. 12 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s Has CDD increased access to education for boys & girls? Photo © World Bank combination of quantitative and qualitative Capacity building for M&E research methods will produce more reliable In addition to having a sound M&E framework, task results. There is a particular need for mixed team leaders (TTLs) need to ensure their program has methods when carefully chosen control groups the capacity to deliver sound M&E. This requires: are not available. 5. Whereas routine monitoring relies on program 1. Building the right capacity through: staff and systems (and is complemented by ex- 77 Hiring some trained staff ternal monitoring), impact evaluation is much 77 Training, both on-the-job and course-based more likely to require outside specialists. This 77 Bringing in outside expertise where needed is because of the techniques required to collect (e.g. to help with impact evaluation) accurate data (household surveys and in-depth 77 Supporting participatory and other external studies on particular topics) and to undertake local monitoring systems. 2. Providing incentives to deliver good M&E through: credible independent analysis. 77 Valuing and using the evidence generated 77 Establishing systems for quality assurance. For Further InFormatIon For Further InFormatIon F. Leeuw and J. Vaessen. 2009. Impact Evaluations and Development: Nonie Guidance on Impact Evaluation. Wash- IFAD. 2007. Managing for Impact in Rural Development: A Guide ington, DC: World Bank, Independent Evaluation Group. Avail- for Project M&E. Rome: IFAD. Available at: http://www.ifad.org/ able at: http://www.worldbank.org/ieg/nonie/guidance.html. evaluation/guide/index.htm. G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 13 Collecting and using the information 77 Think through how the data will be used before Routine monitoring data should be collected by the collecting it. This includes the evidence needs of program management information system (MIS). the program (set out in the RF) and local stake- Indicators in the MIS should be disaggregated by gen- holders; for example, feeding information on der wherever possible. A small number of new, “early changes in service delivery into local decision- warning� gender indicators—for example, on whether making structures. program processes are excluding poor women—should 77 Only collect data that will be used. This is espe- also be part of the MIS. However, the MIS should be cially important where participatory methods relatively simple to work efficiently and so the number are involved, as there are considerable time costs of new gender indicators (i.e. not monitoring indicators for local people. disaggregated by gender) in the MIS will be limited. 77 M&E takes an enormous amount of time and different levels of expertise. Hire specialized as- Gender indicators outside the MIS can be collected by sistance for certain areas such as impact surveys project staff as part of specific studies; for example, look- (Wong 2009). 77 Use trained personnel to moderate and interpret ing at who is benefiting from service improvements. In focus groups. As Luntz (1994) makes clear, “The general, data collected by program M&E staff should be results are dependent upon the interaction be- subject to external quality assurance. This can simply tween the respondents and the moderator, and involve independent annual reviews verifying the fig- unprofessional moderating can lead to inaccu- ures reported for a sample of program sites. rate conclusions.� Many aspects of agency or empowerment (domestic violence or even whether women and men make deci- For Further InFormatIon sions jointly on children’s education, for example) are highly sensitive and data are best gathered in private IFAD. 2007. Managing for Impact in Rural Development: A face-to-face survey interviews by trained female enu- Guide for Project M&E. (Section 6) Rome: IFAD. Available merators. This should be supplemented by qualitative at: http://www.ifad.org/evaluation/guide/index.htm. evidence gathered from focus groups or more special- F. Leeuw and J. Vaessen. 2009. Impact Evaluations and ized techniques such as peer ethnographic review (Price Development: Nonie Guidance on Impact Evaluation. (Part and Hawkins 2005). II) Washington, DC: World Bank, Independent Evaluation Group. Available at: http://www.worldbank.org/ieg/nonie/ General messages on collecting and using M&E evi- guidance.html. dence are: Why do gender process indicators matter? If the program objective is to reduce poverty, there is a need to know that CDD projects reflect the needs of the poor. If local elites decide on project priorities, the program will fail. Likewise, poor men and women may have different priorities and the project need to know that it reflect the needs of both. Photo: Indonesia. © Curt Carnemark /World Bank 15 examples of Gender Indicators for CDD programs 3 T he structure of this section follows the generic CDD logical framework presented in Figure 2 and 1 (Section 2.2 above). We begin with pro- cess indicators and then consider output and outcome indicators. In practice, process indicators are typically Process indicators from program reporting/ management information systems 1. Percent of village development committee mem- bers who are women 2. Are there single-sex groups in the planning pro- placed alongside activity or output indicators in the cess that identify priority projects? results framework (RF). 3. If so, what proportion of projects implemented were proposed by (a) women-only groups, (b) by Whatever sector they are in, CDD programs tend to face men-only groups, or (c) were priorities for both similar process issues. The process indicators discussed groups? here are likewise broadly applicable. Output and out- come indicators are more sector-specific, and although The Lao PDR PRF pilot provides an example where we present examples from programs with a wide range projects are proposed at the village level, but decisions of objectives, they are examples. In some cases, program on which projects to implement are taken at a higher teams will be able to use example indicators directly in local level (i.e. the district). In such cases, it is impor- their RF, but in others, indicators will need to be modi- tant to monitor whether the reason that male or female- fied for the specific sector and local context. favored projects get implemented is due to gender bias at the village or at the district level. Where project iden- Many of the examples presented here reflect indicators that are already in RF but disaggregated by gender. In addition, there are examples of specific gender indica- tors that will add a significant amount of value for both monitoring and evaluation. Program processes These indicators tell you how the program is imple- mented and who is involved. Monitoring and acting on gender process indicators is critical to ensure that CDD projects meet the needs of both men and women. If biases are not fixed at this stage, it is very difficult if not impossible to overcome their effects later on. As dis- cussed in Section 2.1, these monitoring indicators pro- vide essential early warning for managers when there is an opportunity to change practice. Women from the White Hmong ethnic minority Fortunately, there are a number of straightforward pro- group during a focus group discussion about the cess indicators that are based on data that can be col- PRF, Lao PDR. Photo © Andres Engvall / World Bank lected by project field staff. 16 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s project employment to try and minimize the risk of bOx 3 How gender indicators on program process have paid jobs only going to men11 (see Box 4). This illustrates changed practice the importance of disaggregating voluntary and paid employment indicators by gender, but also highlights The Poverty Reduction Fund (PRF) CDD program in Lao PDR uses the value of qualitative investigations regarding pro- single-sex subgroups to generate infrastructure project proposals. Program monitoring identified weaknesses in the original process cesses and local norms, which would give the team an used to select CDD projects, and since 2008/9 the PRF has required idea of how social mobilization and recruitment need that at least two of three infrastructure proposals put forward from to be changed. the combined village meeting come from the women’s list. For CDD programs that provide direct employment, Source: Measuring Impact of Community-Driven Development Projects on Gender—Toolkit for the Poverty Reduction Fund, Lao PDR. World Bank, 2011a. relevant indicators to consider are: 4. The number of men gaining voluntary employ- tification and implementation is separated in this way, ment on program activities in the past year and indicator 3 should be replaced by: over the life of the program. 5. The number of women gaining voluntary em- 3a. If so, what proportion of projects put forward to ployment on program activities in the past year the district were proposed by (a) women-only and over the life of the program. groups, (b) by men-only groups, or (c) were pri- 6. The number of men gaining paid employment on orities for both groups? program activities in the past year and over the 3b. If so, what proportion of projects implemented life of the program. were proposed by (a) women-only groups, (b) by men-only groups, or (c) were priorities for both 11 These are treated as process indicators because they will tell us groups? whether the process used by the program to allocate paid versus voluntary employment disadvantages women. Indicators to monitor The Indonesian PNPM program illustrates the value of total employment generation by CDD programs are discussed under monitoring the gender balance in paid and voluntary output indicators below. Evaluations of major CDD projects in Indonesia and the Philippines have asked: Are paid jobs within the CDD being equally distributed between men and women? Photo: Men carrying bamboo for construction in Indonesia. © Curt Carnemark / World Bank G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 17 7. The number of women gaining paid employment on program activities in the past year and over bOx 4 evidence from gender indicators on project employment the life of the program. In Indonesia, evaluation of the Program Nasional Pemberdayaan Indicators from individual interviews Masyarakat (PNPM) found that traditional attitudes led to women Indicators 1, 2, and 3 provide essential information from taking voluntary project posts, with paid jobs going to men. To ad- project staff on whether the preferred projects of men dress this, the evaluators argued for changing program recruitment and women are the ones actually implemented. The fol- and employment procedures. This also illustrates the value of report- lowing indicators in this subsection provide additional ing on type of project employment by gender. information that is worth collecting if you are using a Source: Gender in Community Driven Development Project: Implications for PNPM Strategy, household survey at baseline, mid-term, or end of proj- Working Paper on the Findings of Joint Donor and Government Mission p. 23. ect. The advantages of these data are that you get: 77 Evidence from (what should be) a statistically representative sample of beneficiaries 77 The opportunity to disaggregate this aspect of Program Outputs and Outcomes governance by socioeconomic group as well as Overview gender; for example, any marginalized group. For convenience, we have grouped CDD interventions into key areas that contribute to well-being: 8. Proportion of men who believe their views influ- ence projects selected at the village level. 77 Access to and quality of services (education, 9. Proportion of women who believe their views health, water & sanitation, credit etc) influence projects selected at the village level. 77 Work and income 77 Time saving (where the direct impact is on wom- Where projects are proposed at the village level but are en but the whole household is affected (Blackden implemented at a higher local level (e.g. district), addi- and Wodon 2006) tional indicators are required: 77 Increased agency or empowerment. 8a. Proportion of men who believe their views influ- Two types of gender indicators are required to monitor ence projects approved for funding at the district and evaluate project interventions in these areas. First, level. there is the simple step of disaggregating key existing 9a. Proportion of women who believe their views indicators by gender to capture how program invest- influence projects approved for funding at the district level. The following type of question can be used to generate Table 3 asking how much influence your views had on data for these indicators: the projects selected for funding I am I don’t Qualitative evidence— certain My views think my typically from focus groups my views were views Qualitative research is very helpful in understanding were probably were what is driving changes in the proportion of men and taken taken taken into into into Don’t women who believe they can influence projects selected account account account know for funding. It can be difficult to interpret changes in Thinking about the projects indicator values without it. If external M&E specialists in your (village/district) are brought in for baseline, mid-point and end of proj- funded by (CDD program ect assessments it is worth using rigorous qualitative name), which of the following best describes 4 3 2 1 research to ask: how much influence your views had on the projects What do women believe are the main factors limiting their that were selected for influence on which projects are actually implemented? funding? 18 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s ment affects access to services, acquisition of assets, program impact. Recommended quantitative indica- and new sources of incomes or improved incomes. The tors12 are: step of disaggregating existing indicators by gender is illustrated in detail for education in the subsection 10. Percent change in female enrollment in primary below. We then show how the same idea can be applied schools in other service sectors. 11. Percent change in male enrollment in primary schools Second, some new gender indicators are required 12. Percent change in female enrollment in second- to capture impacts on time saving and empower- ary schools ment. These are set out—together with the meth- 13. Percent change in male enrollment in secondary ods of collecting this information—in the following schools subsections. 14. Percent change in female completion of primary school Access to services— 15. Percent change in male completion of primary the example of education school The general principle of disaggregating outcome indi- 12 These allow calculation of two World Bank core Indicators at a na- cators by gender is illustrated for education below. In tional level: (1) primary completion rate (PCR) (MDG2) (Tier 1); and this example, the evidence would be used to evaluate (2) Gender parity index (GPI) (MDG3) (Tier 1). Why do gender output and outcome indicators matter? As CDD programs aim to enable communities to improve their well-being, it is important to identify whether and how men and women gain from the program. If up to half of the target group is missing out, the program is failing to deliver. Photo: Indonesia. © Ray Witlin / World Bank G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 19 These indicators capture gender differences in access Access to services—generalising the to the service (via enrollment) but also an aspect of approach to other service sectors whether service improvements affect boys and girls As with education, other service sectors should have equally (via completion rates). Where possible, gender outcome indicators by gender. In terms of access, this differences in both access to services and improvement should include: in services should be captured. For example, program activities may lead to increased female primary enroll- 16. Percent change in access to health services for ment, but when there is a need to ration school fees, men girls do not take end-of-year exams and hence comple- 17. Percent change in access to health services for tion rates fail to rise. women 18. Percent change in access to water and sanitation In order to produce these indicators, you will need facilities for men data just before the project intervention (baseline 19. Percent change in access to water and sanitation data), and then if possible at annual, mid-term, and facilities for women. end of project on: The Indonesian PNPM results framework13 illustrates 77 The number of primary and secondary school how a mixture of government and project annual age girls and boys in the project area. household surveys can capture these indicators, but 77 The number of primary and secondary school also identify that there are certain aspects of health that age girls and boys in any control sites. are particularly relevant for women. These require one 77 The number of boys and girls enrolled in prima- or more female-specific indicators such as: ry and secondary schools in the project area. 77 The number of boys and girls enrolled in pri- 20. Percent increase of girls and women receiving mary and secondary schools in any control health benefits (full immunization; anti-natal, pre- sites. natal; and postnatal care; maternal health care) 77 The number of boys and girls completing pri- mary schooling in the project area and in any This indicator14 indirectly captures whether the quality control sites. of services have changed, as well as access for women. Disaggregating quality of service indicators by gender Data collection issues are discussed in Section 2.5.4. is particularly useful in the health sector, as men and Here we note that this type of data is unlikely to be cap- women can value CDD support for local health centers tured by the program MIS, may possibly be recorded quite differently depending on what kind of services are during specific studies, but is most likely to be collected improved. as part of an evaluation. Household (sample) surveys Some programs are able to capture this information are often used to produce data for these indicators, with from health management information systems (MIS) questions on the age of children in the household and or via local project staff. More generally (and in the case whether they attend or have completed primary or sec- of PNPM), this information is drawn from household ondary school. surveys (relying on women interviewed). Contextual methods (such as focus groups) can add a Perception questions in household surveys also provide great deal of value to the quantitative education indi- the opportunity to directly identify whether beneficia- cators by explaining why indicators are at a relatively ries believe that both access to services and quality of high or low level. Focus group findings on whether services have changed. If used, these should be disag- the project has affected primary and secondary school gregated by gender. As with education indicators in the education of boys and girls differently can often help subsection above, qualitative evidence is very useful in explain changes in the indicator values, even where understanding what has driven the change in indicator these appear counterintuitive. This is well-illustrated values and why these impacts may be different for men by the experience of the Lao PRF program (for indica- and women. tors on time-saving) in Box 6 below. Qualitative findings should be brought out in program reporting alongside 13 For 2010 onwards. the quantitative indicator values. 14 Which could be expressed as separate indicators on immunization etc. 20 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 22. The percent change in working-age men engaged in paid work as a result of the project. Program staff can collect data on the proportion of working-age men and women in paid work as part of a monitoring program, altho ugh it is also desirable to have independent sample survey data at the start, mid- point, and end of the project. The ability to attribute changes in economic indicator values to the project will depend on the M&E tech- niques used by a program; this also has implications for how questions will need to be worded. Some of the major issues are highlighted in Section 2.5.2, together with sources of further information. Here we note that where a control—as well as project (interven- tion) sites—is used, it is much easier to separate out program impacts from broader changes in society or NGO activities that affect whether women gain paid work. In this case, the data required over the project life are simply: Impact evaluation results from the KALAHI-CIDSS project indicate that the project led to a 5 percent increase in women’s labor force participation 77 The percentage of working-age men and women compared to what would have happened without the project. The project engaged in paid work.15 also had a positive but lower impact on men’s labor force participation. Photo: A day-care center funded by the KALAHI-CIDSS project in Municipality Where there are only project sites the same data can La Castellana, Visayas region, Philippines. © Sean Bradley / World Bank be used if outside influences (ranging from govern- ment policy to other project activities) are unlikely to have influenced these indicators. This is only plausible if there are minimal project interventions from govern- The same approach applies to other service sectors not ment or NGOs that affect opportunities for female paid mentioned above, such as credit. work in a project area. Work and income Where such interventions do occur, a second-best When tracking project economic impact, pilot test option is to use a household survey (with a sufficient experience supports the use of straightforward mea- sample of working-age respondents) at the project mid- sures by program M&E teams. Although the Lao PDR and end-points to ask respondents directly how much pilot found that it was possible to identify if female ben- difference the project has made to the amount of paid eficiaries believed their income had risen as a result of work received. An example of this type of question and the project, enumerators nevertheless faced the chal- associated indicator is given in Annex 2. lenge of identifying non-monetary income. Calculating how much income has increased is considerably more No matter which approach is used, qualitative assess- difficult, and the Philippines KALAHI-CIDSS project ment—such as using focus groups—in a sample of sites experience confirms this is unlikely to be an appropri- where the indicator shows significant change is impor- ate indicator (Box 5). In both cases, it proved difficult tant to understand how the project has (or has not) to link changes in income to specific projects. For these succeeded. reasons it is preferable to measure the intermediate out- come gender indicators: 15This has to be measured consistently and can be defined as “any 21. The percent change in working-age women paid work in the past week� (to capture part-time or informal work) engaged in paid work as a result of the project. through to “At least X days of paid work in the past week.� G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 21 bOx 5 measuring economic impact, pilot test evidence Feedback from the pilot test of household survey and FGD instruments underscored the difficulty in generating information for assessing the contribution of KALAHI-CIDSS to the economic empowerment of women. Perhaps some of the problems encountered in generating the desired information stemmed from an inability of the respon- dents to give exact figures about income derived from productive work. Since many project areas are among the poorest in the country, its residents—both women and men—are not fully employed. Many had difficulty comput- ing and recalling income they received from various sources. To address these problems, the team adjusted the questionnaire to enable the respondents to identify all possible sources of income and to provide units of computa- tion (e.g. income earned in a day or in a week) that are more accessible. Some of the variables on productive assets (such as ownership of farm lands or household electrical appliances) were not applicable, again given the low level of economic and educational status of many households in the project areas. Source: Making everyone Count. Gender-sensitive Monitoring and Evaluation in a Community-Driven Development Program: The Case of the Philippines’ KALAHI-CIDSS. World Bank, 2011b. Time saving services tions had saved them time (Box 6). This perception Where CDD projects produce improved access to indicator can be written as: assets—such as drinking water, fuel efficient stoves, threshing machines, new bridges, or local roads—an 23. % of women reporting a reduction of time outcome can be time-saving for women in the house- spent on daily household tasks as a result of hold. Freeing up time from domestic chores can lead the program. to increased economic activity and subsequent pov- erty reduction (Francisco 2007), as well as making it It can be obtained from the type of question in Table 4. easier for women to take an active part in community life (UNIFEM 2008). This can be captured using a small The advantage of asking community members whether number of specific indicators targeted at women and the CDD project had reduced time spent is that they girls. are being asked to think about project-specific impacts. However, to be reliable, respondents need to be It is important to interpret time-saving indicators along- reminded of all the relevant infrastructure investments side indicators in changes in access to service and ser- and have the opportunity to reflect on how these have vice quality. For example, measures of success include impacted on how time is spent. The example in Box 6 both the amount of clean water available and time spent suggests that qualitative research will often be neces- collecting water. sary to explain what this indicator really means. The Lao PDR RPF pilot found it was possible to ask Respondents may find it easier to describe the time women directly whether a range of project interven- saved from one project investment at a time than try- Table 4 asking for estimates of project-specific impact on time spent It has It has It is much It is a little There is no increased a increased less less difference little a lot Thinking about the projects in your (village/ district) funded by (CDD program name), which of the following statements best describes the effect they have had on the 5 4 3 2 1 amount of time women in this household spend on household tasks (e.g. collecting water, fuelwood, etc)? 22 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s bOx 6 using focus group findings to explain changes in indicator values In the pilot test, women who had confirmed their use of or access to the PRF subproject in their villages were asked whether the subproject had changed the time they spend on daily household tasks. They were also given specific examples, such as cooking or collecting water or firewood. The results are interesting and perhaps even counterin- tuitive. On the one hand, one-third of the Lao-Tai women and nearly three of five ethnic minority women reported that they spent less time on household chores as a result of the PRF subproject. On the other hand, about one-third of all women reported that the time spent on household chores had increased despite the PRF subproject. To understand why this situation occurred, the pilot test included several focus group discussions. These discussions indicated that educational subprojects can affect the time devoted to household chores. When older children attend school, the burden on their mothers can increase because they must assume responsibility for taking care of infants, gathering firewood, or other tasks formerly handled by their older children. Source: Measuring Impact of Community-Driven Development Projects on Gender—A Toolkit for the Poverty Reduction Fund, Lao PDR. World Bank, 2011. ing to estimate the joint impact of all project inter- it saves them. If control sites are used, just asking about ventions (e.g. have a question on time saved from time taken will give a very accurate picture of time saved improved water access followed by a separate question as a result of the project; that is, the change in time on time spent collecting fuelwood). In this case, both spent in the project area compared with the change in sector-specific and a composite time-saving indicators time spent in the control area. The disadvantage of this can be reported. approach is apparent where there are no control sites, as the time taken to access the service may vary signifi- Evidence on this aspect of CDD project impact can also cantly by season or due to external factors (for example, be obtained by asking women at a project meeting— government policy changes, other projects), so changes during planning before, for example, a water project is in time spent on this task do not only reflect the project implemented, and then approximately a month after it in question. has been in operation—the following question: How many minutes does it take you to go and fetch water Note that for both approaches described in this sub- and return? It is also important to identify the propor- section, collecting information on time spent requires tion of women who use the new or improved service; for training project staff or survey enumerators to use example, to check whether any time saving is confined common units of measurement (e.g. nearest half hour to the elite). or minutes). In the relatively rare instances where com- munities do not record time spent on household tasks, The same approach can be used for other projects that any indicator of time saved based on recollection will be aim to save women’s time; for example, fuel efficient inaccurate. A solution that has been used in some CDD stoves, threshing machines, or possibly new bridges or projects is to have trained personnel (who can be from roads. This will allow calculation of the following indi- the local community) monitor the time taken to access a cators for each project: particular service for a representative sample of women in the community. 24. The proportion of women who use the new asset or service. The following flow chart will help you decide which 25. Average time saved per day for each woman us- indicators of time saving to use, together with the ing the new asset or service. appropriate method of data collection. There are pros and cons to this approach. One advan- Increased agency or empowerment tage is that just asking about time taken for accessing a This subsection focuses on identifying whether a CDD particular service means it is much less likely that a per- project significantly affects the ability of individuals to son’s positive or negative views toward a CDD project make choices and take actions within the household in general will influence the answer of how much time and in the broader community. As women in the EAP G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 23 Region currently face major constraints in terms of ized techniques such as peer ethnographic review (Price decision making (World Bank 2010d), the indicators and Hawkins 2005). in this subsection focus primarily on attitudes toward female decision making in economic, social, and politi- Questions to ask both men and women cal arenas. The Indonesia PNPM and Lao PRF gender indicator pilots highlight capturing the impact of the program on Many aspects of agency or empowerment—for exam- female empowerment in terms of the: ple, domestic violence or even whether women and men make decisions jointly on children’s education— 77 Change in attitude of men and women regarding are highly sensitive and data are best gathered in private women’s roles in social, political, and economic face-to-face survey interviews by trained female enu- activities (PNPM). merators. This should be supplemented by qualitative 77 Perceptions of women’s roles in decision making evidence gathered from focus groups or more special- (PRF). fIGure 4 Which method to use to generate indicator data Use questions on time taken for speci c household tasks & compare change pre- & post-project intervention for project & control sites Can respondents report time spent on Yes Are project Yes accessing services/household & control sites tasks with reasonable available? accuracy? No No Use trained local community members, Is it likely that since the project staff, or consultants to record the project investment other factors time taken on key household tasks by (e.g. seasons, policies or other projects) representative women before & after could have signi cantly in uenced project intervention the time taken for the household tasks in question? Yes No Ask respondents to Use questions on time taken consider the speci c for speci c household tasks effects of this project on & compare change pre- & each household task & post-project intervention for report the time saved project sites 24 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s One major issue that persists throughout the region in countries at all stages of development is that women’s voice in formal decision-making remains weak. Photo: China © Curt Carnemark / World Bank Outcome indicators can either seek to capture changes Yet the experience of the Philippines KALAHI-CIDSS in attitudes as a result of the program, or look for pilot was that men were often unable to answer survey changes in behavior as a result of the program (reflect- questions on what they referred to as “women’s affairs� ing both changes in attitudes and assets). (Box 7). In this type of situation, qualitative investiga- tion—rather than using a survey questionnaire—is The evaluation of Afghanistan’s National Solidarity Pro- likely to be the most effective way of getting men to dis- gramme (NSP)—see Annex 1—provides examples of cuss these issues. capturing male attitudes toward female empowerment with two broad categories of indicators:16 As many CDD M&E teams face capacity constraints, we suggest asking a simple outcome indicator for political 1. Attitudes toward female participation in local empowerment as follows: governance. 2. Attitudes toward female employment and 26. The percent change in local female elected offi- community life. cials (village and municipal offices) as a result of the project. Some of the specific indicators within the “Female Socialization and 16 Mobility� category reflect the highly conservative nature of society in It is straightforward for project staff to obtain figures Afghanistan and are not transferable to other contexts. on the proportion of elected local officials who are G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 25 female from local government at the start, mid-point, and end of a project. The challenge is to attribute any bOx 7 Identifying male attitudes to changes to the project with confidence; this is dis- female empowerment: evidence from cussed further below. the philippines One of the key findings of the field test was the dif- The ability to attribute changes in political empower- ficulty in getting the male respondents to share in- ment indicator values to the project will depend on the formation and their opinions about women, includ- M&E techniques used by a program. The techniques ing their spouses. While some said they had little available also have implications for how questions will knowledge about the activities and opinions of their need to be worded. wives, others were hesitant to respond to the inter- view questions for the simple reason that they are Where control as well as project (intervention) sites are “not used to talking about women’s affairs.� used, it is much easier to separate out program impacts Source: Making everyone Count. Gender-sensitive Monitoring and Eval- from broader changes in society or NGO activities that uation in a Community-Driven Development Program: The Case of the affect whether women stand in local elections. In this Philippines’ KALAHI-CIDSS. World Bank, 2011b. case, the data required over the project life is simply: 77 The percentage of elected officials (village and project has increased the number of elected female municipal offices) that are women. local officials (see table 5). Where there are only project sites, the same data can No matter which approach is used, qualitative assess- be used if outside influences (ranging from govern- ment policy to other project activities) are unlikely to ment (e.g. using focus groups) in a sample of sites have influenced these indicators. This is only plausible where the indicator shows significant change is if there are minimal project interventions from govern- important to understand how the project has suc- ment or NGOs that affect female empowerment in your ceeded (or not). project area. Questions specifically directed at women Where such interventions do occur, you will need to use There are many examples from the international lit- a household survey (with a sufficient sample of working erature of indicators that capture the ability of women age respondents) at the project mid- and end-points to to exercise choice over social, economic, and political ask respondents directly how much difference the proj- dimensions of life. The challenge is to identify indica- ect has made to the election of women local officials. As tors that will produce reliable results for program M&E respondents typically find it difficult to give precise quan- teams under typical field conditions. For example, both tities (for the reasons set out in Box 7), these questions the Lao PDR PRF and Philippines KALAHI-CIDSS need to cover broad categories of impact. For example: pilots identify the need to capture women’s roles in household decision making. Based on questions used With this type of question, the indicator has to be slightly in a randomized control trial of the impact of a personal modified to reflect the possible responses and might be financial savings account on women (Ashraf et al. 2006), written as: The percent of women who believe that the the KALAHI-CIDSS specifies: Table 5 asking for estimates of project-specific impact on election of female officials It has It has It is much It is a little There is no increased a increased less less difference little a lot Thinking about the projects in your (village/ district) funded by (CDD program name), which of the following statements best 1 2 3 4 5 describes the effect they have had on the amount of elected female local officials compared to the situation without the project? 26 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s The percentage of women making decisions regarding: Which statement best describes your power We a. what to buy at the market to decide on the It is my decide I am not b. asset purchases following: decision jointly involved c. number of children A) What to buy at the d. schooling of children market e. use of family planning. B) Asset purchases C) Number of children The Ashraf et al. (2006) study this question is drawn D) Schooling of from is able to define “decision making� by women children precisely as it refers to use of funds held in a personal E) Use of family savings account. In general, however, it is difficult to planning specify exactly what “making decisions� means; that is, it can mean solely or jointly with other household The indicator can then be defined using any of the three members. Inconsistent responses across areas or over response categories, but the following is perhaps the time will produce misleading results. For wider CDD most powerful: project use, this indicator should be based on a ques- tion of the form: 27. The percentage of women who report having no power to make decisions regarding: a. what to buy at the market b. asset purchases c. number of children d. schooling of children bOx 8 Choosing key empowerment indicators e. use of family planning. Based on a review of empowerment indicators used in Bangladesh Three other core empowerment indicators have been and Ethiopia and qualitative research in project sites, three key in- drawn from the literature (Box 8). These are: dicators were chosen to assess the empowerment impact of micro- credit for poor women in India: 28. The percentage of women who know how much income and expenditure there is in the house- • Percent of women who know how much income and expenditure hold. there is in the household. 29. The percentage of women who can travel outside • Percent of women who can travel outside the village to visit rela- the village to visit relatives without permission. tives without permission. 30. The frequency of domestic violence experienced • Frequency of domestic violence experienced by women in the past by women in the past year. year. Data for these indicators can potentially be collected This information was collected by community animators, project- by trained local female program staff or female survey trained individuals who link self-help groups (SHG) to the NGO enumerators brought in as part of the team administer- providing credit. The community animators work closely with the SHG, and women from the community were comfortable discussing ing a baseline, mid-term, and end-of-project survey. these issues with them. The project impact has been evaluated first by asking women to compare the situation for these indicators now As we have seen in previous sub-sections, it is chal- and before the SHG was formed. Newly formed SHGs are also being lenging to attribute changes in indicator values to the compared with control groups. project when there are no control sites. One option is to ask respondents directly to consider how the project An unexpected challenge has been that once project staff had care- has changed attitudes or behavior. The other is to argue fully explained the purpose of the survey in control villages, gaining that any change in attitudes can only realistically reflect sufficient trust to get sensible replies for these indicators, they faced the influence of the project. As norms and attitudes in an almost overwhelming demand to bring the project to the control society as a whole tend to change over longer periods sites. They have promised to do this within 18 months. Source: Yaron, Choudhary, and Best 2008. G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 27 An outcome indicator related to measuring empowerment could be: Percentage of women who know how much income and expenditure there is in the household. Photo: Vietnam © World Bank of time17 than access to assets or services, it is probably the indicators to be modified in exactly the same way safer to attribute observed changes in attitudes to the described in relation to Table 5 above. project than it is to attribute changes in other areas. Despite this general observation, if it is likely that activi- Summary of example indicators ties by other projects will affect the answers given by The summary of example indicators below uses the participants in a project, survey managers will need to same indicator numbering as used elsewhere in Sec- ask respondents directly how much difference the proj- tion 3. It is further divided into the areas of the generic ect has made to each of the indicators. This will require logical framework that we have used to structure this toolkit. Indicators in the “additional indicators� section 17The evidence for Bangladesh, for example, points to intergenera- may be very important for some programs and are given tional change (World Bank 2007b) rather than the one-to-two-year period of involvement with social safety net programs (Yaron and this title simply because other indicators can provide Dudwick 2008). related information. 28 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s Gender indicator examples (as numbered in the document text) Process indicators 1. Percent of village development committee members who are women 2. Are there single-sex groups in the planning process that identify priority projects? 3. If so, what proportion of projects implemented were proposed by (a) women-only groups, (b) by men-only groups, or (c) were priorities for both groups? Output & Outcome indicators—Improved services (education) 10. Percent change in female enrollment in primary schools 11. Percent change in male enrollment in primary schools 12. Percent change in female enrollment in secondary schools 13. Percent change in male enrollment in secondary schools 14. Percent change in female completion of primary school 15. Percent change in male completion of primary school Output & Outcome indicators—Improved services (health & water & sanitation) 16. Percent change in access to health services for men 17. Percent change in access to health services for women 18. Percent change in access to water and sanitation facilities for men 19. Percent change in access to water and sanitation facilities for women 20. Percent increase of girls and women receiving health benefits (full immunization, anti-natal, and prenatal and postnatal care, maternal health care) Output & Outcome indicators—Increased incomes 21. The percent change in working-age women engaged in paid work as a result of the project 22. The percent change in working-age men engaged in paid work as a result of the project Outcome indicators—Improved services (time-saving) 23. Percent of women reporting a reduction of time spent on daily household tasks as a result of the program 24. The proportion of women who use the new asset or service 25. Average time saved per day for each woman using the new asset or service Outcome indicators (empowerment) 26. The percent change in local female elected officials (village and municipal offices) as a result of the project 27. The percentage of women who report having no power to make decisions regarding (a) what to buy at the market; (b) asset purchases; (c) number of children; (d) schooling of children; and (e) use of family planning 28. Percentage of women who know how much income and expenditure there is in the household 29. Percentage of women who can travel outside the village to visit relatives without permission 30. Frequency of domestic violence experienced by women in the past year* additional gender indicators Process indicators 4. Number of men gaining voluntary employment on program activities in the past year and over the life of the program 5. Number of women gaining voluntary employment on program activities in the past year and over the life of the program 6. Number of men gaining paid employment on program activities in the past year and over the life of the program 7. Number of women gaining paid employment on program activities in the past year and over the life of the program 8. Proportion of men who believe their views influence projects selected at the village level 9. Proportion of women who believe their views influence projects selected at the village level *Based on four categories: (a) Zero, (b) 1-2/year, (c) 3-5/year, and (d) 6+/year. 29 annexes Annex 1. Evaluating the NSP in Afghanistan The National Solidarity Programme (NSP) is the larg- Group Indicator est development program in Afghanistan and a flagship program of the government. It is structured around Supports Female Participation In Local Government two major village-level interventions: (1) the creation Supports Women’s Council for Consultation Attitudes of a gender-balanced Community Development Coun- toward Female Supports Women’s Council for Female Affairs cil (CDC) through a secret-ballot, universal suffrage Participation election; and (2) the disbursement of grants to support in Local Does Not Support Any Female Involvement the implementation of projects selected and managed Governance (asked to both Women Should Participate in Elections by village communities. NSP so far has established men & women) Women Should Help Select Headman approximately 22,300 Community Development Coun- cils (CDCs) across 361 districts in all of Afghanistan’s 34 Women Should Help Select Governor provinces. It has financed over 51,000 projects involving water and sanitation, rural roads, electrification, irriga- Agrees with Women in Government Attitudes tion, and human capital development. toward Female Agrees with Women in NGOs Employment and Respect Agrees with Girls Attending School The randomized impact evaluation of the Phase-II of NSP (2007–10) is a multiyear study designed to assess the Accorded Male Doctors Should Treat Women Women (asked effects of the program across a broad range of economic, to both men & Respondent Places Equal Value on Female Births institutional, and social indicators. This report presents women) interim estimates of these effects obtained using data col- There are Respected Women in Village lected from over 20,000 individuals in 500 sample villages Meetings with Women from Other Villages immediately before the introduction of NSP (baseline survey in summer 2007) and again two years later (first Meetings with District Government follow-up survey in summer-autumn 2009). Female Women Have Source for Counseling Socialization and Socializes with Women Outside Family The survey tracks outcome indicators in a number of Mobility (asked areas, including the following indicators of attitudes only to women) Number of Times Outside Compound toward female participation in local governance, employ- Leaves Compound without Chaperone ment, and community life as well as opportunities for socializing and local travel. Does Not Always Wear Chadori Source: Beath, Christia, Enikolopov, and Kabuli 2010. 30 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s Annex 2. Asking about perceived impact on paid work As respondents typically find it difficult to give precise quantities when asking about differences made by a CDD project (for the reasons set out in Box 5 ), these questions need to cover broad categories of impact. For example: asking for estimates of project-specific impact on paid work It has It has It is much It is a little There is no increased a increased less less difference little a lot Thinking about the projects in your (village/ district) funded by (CDD program name), which of the following statements best 1 2 3 4 5 describes the effect they have had on the amount of paid work you have received compared to the situation without the project? With this type of question the indicator has to be slightly modified to reflect the possible responses and might be written as: The percentage of working-age women who believe that the project has increased their paid work. G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s 31 references Ashraf, N., D.S. Karlan, and W. Yin. 2006. Female Garabino, S., and J. Holland. 2005. “Quantitative and Empowerment: Impact of a Commitment Savings qualitative methods in impact evaluation and measur- Product in the Philippines. Yale University Economic ing results.� Available at: http://www.gsdrc.org/docs/ Growth Center Discussion Paper No. 949; Center for open/EIRS4.pdf. Global Development Working Paper No. 106. Available at SSRN: http://ssrn.com/abstract=952784. Hentschel, J. 1999. “Contextuality and data collection methods: A Framework and application to health service Barker, C. 2005. Cultural Studies: Theory and Practice. utilization.� The Journal of Development Studies 35: 64–94. London: Sage. Holland, J., and J. Campbell, eds. 2005. Methods in Beall, J. 2005. “Decentralizing Government and Central- Development Research: Combining Quantitative & izing Gender in Southern Africa: Lessons from the South Qualitative Approaches. London: Intermediate Tech- African Experience.� Report No. 8. Geneva: United nology Group. National Research Institute for Social Development. IFAD. 2002. Managing for Impact in Rural Develop- Beath, Andrew and F. Christia, R. Enikolopov. 2010. ment: A Guide for Project M&E. Rome: IFAD. Available Randomized Impact Evaluation of Afghanistan’s at: http://www.ifad.org/evaluation/guide/index.htm. National Solidarity Programme (NSP): Interim Estimates of Program Impact from First Follow-Up IFAD. 2004. Community Development Funds in IFAD Survey. Washington, DC: World Bank. Projects: Some Emerging Lessons for Project Design. Rome: International Fund for Agricultural Development (IFAD). Blackden, C. M., and Q. Wodon. 2006. “Gender, Time Use, and Poverty in Sub-Saharan Africa.� World Bank IFAD. 2006. “Initiative on Community Driven Develop- Working Paper 73. Washington, DC: World Bank. � ment Regional Workshop Report. Accra, FIDAFRIQUE program, IFAD. Also available at: http://fidafrique.net/ Chattopadhyay, R., and E. Duflo. 2001. Women as Pol- rubrique282.html. icy Makers: Evidence from A India-Wide Randomized Policy Experiment. NBER Working Paper 8615. Econo- Jorgensen S. 2005. Framing the logic of CDD operations: metrica 72, 5 (2004): 1409–1444. Toward a project theory for CDD design and evaluation. Washington, DC: World Bank. Dongier, Philippe and Julie Van Domelen, Elinor Ostrom, Andrea Ryan, Wendy Wakeman, Anthony Kusek, J. and R. Rist. 2004. Ten Steps to a Results-Based Bebbington, Sabina Alkire, Talib Esmail, Margaret Pol- Monitoring and Evaluation System: a handbook for ski. 2002. “Community Driven Development� in: Klug- development practitioners. Washington, DC: World man, Jeni. 2002. “A Sourcebook for Poverty Reduction Bank. Strategies�. Washington DC. World Bank. Leeuw, F., and J. Vaessen. 2009. Impact Evaluations and Francisco, J. S. 2007. “Gender Inequality, Poverty and Development: Nonie Guidance on Impact Evaluation. Human Development in South East Asia.� Development The Network of Networks on Impact Evaluation. Wash- 2007 50(2): 103–114. ington, DC: IEG. 32 G e n d e r d i m e n s i o n s o f c o m m u n i t y- d r i v e n d e v e l o p m e n t o p e r at i o n s Luntz, F. 1994. Voices of Victory, Part I. Focus Group World Bank. 2005b. “The Results Chain in a Bank- Research in American Politics. Appeared in the Polling Supported CDD Intervention.� Washington, DC: World Report May 1994. Available at: http://www.pollingreport. Bank. com/focus.htm. World Bank. 2007a. “Enabling East Asian Communities OECD-DAC. 2002. “Glossary of Key Terms in Evaluation to Drive Local Development: East Asia Region CDD and Results Based Management.� Paris. OECD-DAC. Flagship Report.� Unpublished mimeo. Washington, DC: World Bank. Osborne, D., and T. Gaebler. 1992. Reinventing Govern- ment. Boston, Mass.: Addison-Wesley Publishing. World Bank. 2007b. Whispers to Voices: Gender and Social Transformation in Bangladesh. Washington, DC: Price, N.L., and K. Hawkins. 2005. “The Peer Ethno- World Bank and AusAid. graphic Method in Health Research: Some Theoreti- cal & Methodological Reflections.� In J. Holland and World Bank. 2007c. Gender in Community Driven J. Campbell, eds. Methods in Development Research: Development Project: Implications for PNPM Strategy. Combining Quantitative & Qualitative Approaches. Working Paper on the Findings of Joint Donor and Gov- London: Intermediate Technology Group. ernment Mission, World Bank, AusAID, ADB, DFID. World Bank, Jakarta. UNIFEM. 2008. Making the MDGs Work for All: Gen- der-Responsive Rights-Based Approaches to the MDGs. World Bank. 2010a. Indonesia: PNPM M&E Initiative— Bangkok.UNIFEM East and Southeast Asia Regional List of Gender Indicators. Washington, DC: World Bank. Office. World Bank. 2010b. Development for All—A Gender W.K. Kellogg Foundation. 2004. W.K. Kellogg Foundation Action Plan for the East Asia and Pacific Region FY11– Logic Model Development Guide. Available at: http:// 13. Washington, DC: World Bank. ww2.wkkf.org/DesktopModules/WKF.00_DmaSupport/ ViewDoc.aspx?fld=PDFFile&CID=281&ListID=28& World Bank. 2011a. Measuring Impact of Community- ItemID=2813669&LanguageID=0. Driven Development Projects on Gender—Toolkit for the Poverty Reduction Fund, Lao PDR. Washington, DC: Wong, S. 2003. “Indonesia Kecamatan Development World Bank Program: Building a Monitoring and Evaluation System for a Large-Scale Community-Driven Development World Bank. 2011b. Making everyone Count: Gender- Program. EASES Discussion Paper. Washington, DC. � sensitive Monitoring and Evaluation in a Community- World Bank. Driven Development Program: The Case of the Philippines’ KALAHI-CIDSS. Washington, DC: World Wong, S. 2009. “Monitoring & Evaluation For CDD Bank. Operations: Lessons Learned.� Washington, DC: World Bank. Available at: http://www.adb.org/Documents/ Yaron, G., and N. Dudwick. 2008. “Measuring empow- Events/2009/CDD-Rural-Poverty-Alleviation/S361- erment: A Mixed Method Diagnostic Tool for Mea- Susan-Wong.pdf. suring Empowerment in the Context of Social Safety Net Provision in Bangladesh.� Paper presented at UN World Bank, FAO, and IFAD. 2009. Gender in agricul- WIDER “Frontiers of Poverty Analysis� conference, Hel- ture sourcebook. Washington, DC: World Bank. sinki, October. World Bank. 2005a. The Effectiveness of World Bank Sup- Yaron, G., S. Choudhary, and J. Best. 2008. “Impact port for Community-Based and -Driven Development: evaluation of Rojiroti—first round, Research into Use An OED Evaluation. Washington, DC: World Bank. Programme.� London. DFID. THE WORLD BANK Social Development Unit Sustainable Development Department East Asia and Pacific Region 1818 H Street, N.W. Washington, D.C. 20433, USA Telephone: 202 473 1000 Facsimile: 202 522 1666 Website: www.worldbank.org/eapsocial