Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) Report Number : ICRR0020584 1. Project Data Project ID Project Name P106332 BD: Disability and Children at Risk Country Practice Area(Lead) Bangladesh Social Protection & Labor L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-44810 31-Dec-2014 35,000,000.00 Bank Approval Date Closing Date (Actual) 01-Jul-2008 30-Jun-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 35,000,000.00 0.00 Revised Commitment 21,155,430.87 0.00 Actual 18,008,686.78 0.00 Prepared by Reviewed by ICR Review Coordinator Group Erik A. Bloom Soniya Carvalho Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the Financing Agreement, the original objective was "to expand the coverage, use, and quality of social care services for persons with disabilities and vulnerable children as a means of promoting equity and social inclusion." The PAD (p. 4) defined "social care services" as services other than income transfers that address a wide range of human development needs, including "early child development, detection, prevention, physical and social rehabilitation, foster child care, assisted living, career development, and equalizing opportunities for the disabled, services for vulnerable children, etc." It defined "promoting inclusion" as achievable through "awareness campaigns, policy formulation and implementation, networking, Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) community mobilization and empowerment," to "eliminate physical, social, cultural and environmental barriers, which prevent the target groups from participating in society." At a June 2011 restructuring, the objectives were revised to: "to expand coverage and improve quality of services for persons with disabilities and children at risk by strengthening and scaling up government service delivery systems" (Restructuring Paper, May 26, 2011). At that point, 10.6% of total Bank financing had been disbursed. A December 2013 restructuring revised some outcome indicators. This revision involved dropping some indicators and redefining or adding others, impacting three of the project's 13 key outcome indicators. In cases where targets were revised for indicators that were redefined or remained the same, both the original and revised targets were achieved. Current restructuring guidance does not call for a split rating in this instance. This validation will therefore perform only one split. Under the original objectives, the sole assessed objective will be "promote equity and social inclusion," as the goals of expanded coverage, use, and quality of social care services are expressed as outputs intended to lead to that outcome. Under the revised objectives, achievement of two objectives will be assessed: "expand coverage of services for persons with disabilities and children at risk," and "improve quality of services for persons with disabilities and children at risk." b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 24-Jun-2011 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The project originally contained three components: 1. Expanding disability services, improving awareness, and building capacity (appraisal, US$ 22 million): This component was to expand social and rehabilitation services for people with disabilities and build capacity of non-governmental organizations (NGOs), disabled people's organizations (DPOs), and government agencies that support disabled people and their families, and/or that work for greater disability awareness, inclusion, and empowerment. It was to support the restructuring and scaling up of the Jatiyo Protibondhi Unnayan Foundation (JPUF), a national foundation whose goal is to support disability services and awareness, and to channel grants to DPOs and disability NGOs. There were two subcomponents: (a) restructuring and capacity building of the JPUF, to reform and strengthen it into a center of excellence for Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) contracting disability services, promotion/awareness building, research, monitoring and evaluation (M&E), and establishment of good practices; and (b) provision of disability services through NGOs/DPOs, including community-based rehabilitation, early detection and intervention, prevention, counseling, communication and awareness, empowerment, education, skills development, provision of assistive services, and mainstreaming. Windows for financing were to remain flexible and be determined by the JPUF Board. 2. Strengthening and expanding the network of services for children-at-risk (appraisal, US$ 9.5 million): This component was to improve access to, and the quality and relevance of, services for children- at-risk in selected cities. It was to align the country's legal and policy framework with the United Nations Convention on the Rights of the Child (UNCRC), strengthen capacity of the Department of Social Services (DSS) to design and oversee services for child protection, and implement pilot interventions to inform the child protection policy framework. There were four subcomponents: (a) support institutional and capacity building of DSS staff and service providers, providing training and capacity development to enhance the quality of services for children-at-risk; (b) improve and expand services for street children, supporting the expansion of the UNICEF-financed "Protection of Children-at-Risk" (PCAR) project from six to eleven locations; (c) support the DSS to develop and pilot alternative approaches to institutional care for orphans in line with the UNCRC, specifically foster care; and (d) project management, covering the day-to-day management and operational costs of the project. 3. Institutional strengthening of the Ministry of Social Welfare (MOSW) and Department of Social Services (appraisal, US$ 3.5 million). This component was to support a shift of the MOSW and DSS from being managers of institutions and services to a proactive and preventive social service approach. Four key strategic areas were to be supported: (a) strengthening MOSW's capacity for policy formulation and revision of relevant legal frameworks; (b) supporting the institutional development of the DSS by streamlining its organizational structure and procedures; (c) providing training and capacity to enhance the ability of DSS personnel and other service providers for proactive services to target beneficiaries; and (d) strengthening program planning, implementation, and M&E capacity of both MOSW and DSS. At the 2011 restructuring, the components were revised to take into account a major shift in the government's social welfare policy (that led to the restructuring). In early 2009 (following a change of government after elections in December 2008), the government decided to significantly strengthen public social welfare institutions and service delivery systems, and to scale up publicly provided services, while retaining a commitment to partnership with non-governmental providers where necessary to complement publicly provided services. As a result, the JPUF began a process of conversion to a government agency (2011 Restructuring Paper, p. 2, para 3; ICR, p. 6, para 17), with the appointment of a civil servant seconded from MOWS as Managing Director. Services provided by the project were, at this restructuring, envisaged to cover significantly more beneficiaries across the country than originally planned; longer-term systems for service delivery were to be established; the scope and coverage of training and capacity building activities were to be expanded to include nodal government agencies; and new activities were added to assess the socioeconomic situation and needs of persons with disability and children-at-risk. The revised components were: Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) 1. Institutional strengthening, expanding disability services, improving awareness, and building capacity (new proposed cost, US$ 22 million): Similar to component 1 before the restructuring, this component was envisioned to strengthening the capacity of JPUF to formulate and implement disability policies and programs, increase the coverage and quality of disability services and programs, and build public awareness on the rights of persons with disabilities. The two subcomponents were designed to better fit the model of JPUF as a government agency: (a) expansion of integrated services and support for inclusion of persons with disability, intended to strengthen and scale up the existing network of government programs and interventions through the addition of 20 new Care and Assistance Centers for Persons with Disabilities/Disability Service Centers (DSCs); support contract services from non-governmental service providers to reach persons with disability specifically in underserved geographical/service areas in order to complement existing government service delivery; design a national disability certification and determination process through the development of relevant guidelines and tools; provide schooling materials such as Braille textbooks, assistive devices such as eyeglasses and hearing aids, and mobile ramps for cyclone shelters; and design and implement public education and information campaigns on disability issues; and (b) institutional strengthening of the JPUF, including conducting of a socioeconomic assessment of persons with disability and carrying out a needs assessment to inform program development and action plan formulation; developing detailed operational guidelines and instructions as well as standards of service for persons with disabilities; introducing a system to monitor and assess JPUF progress and performance with an integrated management information system (MIS); providing training to disability services providers; conducting an evaluation of JPUF programs and services; and financing fiduciary and technical staff to support project implementation. 2. Strengthening and expanding the network of services for children-at-risk and strengthening DSS (new proposed cost, US$ 10.5 million). This component contained two subcomponents: (a) strengthening services for children-at-risk, including support for Integrated Child Protection Service Centers (ICPSCs) in seven divisional cities, including 24-hour crisis management services; contracting of service provision from non-government service providers to expand services specifically in underserved geographical or service areas for children-at-risk to complement government service delivery; conducting of a socioeconomic situation assessment of children-at-risk, as well as a needs assessment to inform program development and action plans; developing detailed operational guidelines, instructions, and standards of service for children-at-risk; and designing and implementing public education and information campaigns; and (b) project implementation support, including design and implementation of a system to monitor and assess DSS progress and performance, including an MIS; conducting of an evaluation to assess performance of programs and services for children-at-risk; and financing of fiduciary and technical staff to support project implementation. 3. Institutional strengthening of MOSW to mainstream disability and children-at-risk issues (new proposed cost, US$ 2.5 million), intended to support strengthening the National Social Service Academy on disability and child protection issues; building the capacity of MOSW personnel through training and exposure through group study tours; training and building capacity of staff at local levels; developing and implementing an MIS for MOSW integrated with the DSS and JPUF MIS systems; and conducting an evaluation to assess overall project performance and impact. Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) At the 2013 restructuring, the components were again revised to expand core project activities, specifically delivery and outreach services to further increase coverage and quality of services for target beneficiary groups. Some activities that were not considered central to achieving the projects objectives were cut. The revised components and their revisions were: 1. Strengthening disability services (new proposed cost, US$ 16.2 million) was to establish around 30 additional DSCs at the sub-district level and provide additional mobile therapy vans, as well as upgrade equipment in all DSCs. It dropped the activities to support need-based initiatives in underserved geographical/service areas through NGO providers and to provide schooling materials and assistive devices, as these activities no longer reflected government priorities and plans. The provision of support for the design and pilot of a national disability certification system was also dropped, as adequate funds for this activity were budgeted by the government. Finally, the initial plan to develop separate MIS for JPUF and DSS was altered in favor of a consolidated MIS to be implemented under the second component. 2. Strengthening children-at-risk services and social welfare program administration (new proposed cost, US$ 10 million). This component merged the previous second and third components, as both were administered by DSS. It was to establish approximately four additional ICPSCs to expand coverage and provision of additional equipment for strengthening emergency and outreach services. It dropped the activities to support need-based initiatives in underserved geographical/service areas through NGO providers for children-at-risk, as these activities no longer reflected government priorities and plans. It intended to consolidate all MIS activities into an integrated system that covered not only project activities but also key programs administered by the MOSW, including social pensions and cash transfer programs such as the Old Age Allowance, Widows Allowance, and Disability Allowance. This component was to continue the capacity building and institutional strengthening activities formerly under the third component. At a third restructuring in March of 2015, the content of the components remained unchanged, but the proposed costs were reduced to US$ 11.5 million for component one, and US$ 5.7 million for component two. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost: Total project costs were originally estimated at US$ 35 million. The 2011 restructuring shifted funds among the revised components, but total estimated costs remained at US$ 35 million. At the 2013 and 2015 restructurings, cancellation of funds resulted in smaller allocations to each of the two remaining components. Final total project cost was US$ 17.88 million. Financing: The project was to be financed by a US$ 35 million Credit from the International Development Association. US$ 8.8 million was cancelled at the 2013 restructuring, at the request of the government. A further US$ 5 million was cancelled at the 2015 restructuring. At that time, it was also recognized that there had been approximately US$ 4 million in exchange rate fluctuations, such that the Credit amount at that time was US$ 17.2 million. Actual financing was US$ 17.88 million. Borrower contribution: No borrower contribution was planned or made. Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) Dates: The project was approved on July 1, 2008 and became effective on January 22, 2009. A June 24, 2011 level-I restructuring revised the objectives and activities to reflect a shift from NGO-based to government provision of services. A December 31, 2013 restructuring cancelled US$ 8.8 million of project funds (due to delays in initiating services resulting from lengthy government approval processes, cost savings in operational costs of DSCs and ICPSCs, lower-than-anticipated costs of some consulting contracts, and elimination of some activities), modified components and costs, adjusted the results framework, and extended the closing date from December 31, 2014 to June 30, 2016, to allow for the additional DSCs and ICPSCs established under the project to become fully operational and for the integrated MIS to be developed and rolled out at all sites. A final restructuring on March 25, 2015 cancelled US$ 9 million of project funds, partly to eliminate amounts not needed for completion of remaining activities, and partly to reflect exchange rate fluctuations. The project closed on June 30, 2016. 3. Relevance of Objectives & Design a. Relevance of Objectives At appraisal, despite the existence of programs to support many specific marginalized groups, the government of Bangladesh recognized that persons with disabilities and vulnerable children (street children, children working in hazardous conditions, orphans, children/adolescents in conflict with the law, etc.) benefited from few targeted interventions (PAD, p. 1). It was estimated the exclusion of these people from formal social and economic participation, as well as the foregone paid employment of family members in order to provide care, was translating into gross domestic product losses of 5% to 7%. The exact number of children-at-risk was not known, but almost 700,000 children were estimated to be homeless, and 4 million children ages 5-17 were neither in school nor engaged in work. The government had created a National Policy for Disability in 2005 that focused on data collection, development of prioritized district-level actions, public awareness campaigns, improved nutrition services for pregnant women and other measures to strengthen early detection and intervention, skills development and training for disabled persons, and increased access to transport, education, and public facilities. Similarly, in 2005 the government created a National Plan of Action for Children (2005-2010) with objectives to create opportunities for children to reach their full potential, serve the best interests of children in all situations, ensure safety and security of children at home and in public spaces, and establish and protect children's rights. The project's objectives supported each of the pillars of both national strategies. By 2008, Bangladesh was signatory to most international conventions and declarations on disability and children, including the United Nations Conventions on the Rights of Persons with Disabilities and the Rights of the Child. The project's objectives were also relevant to the Bank's Country Assistance Strategy (CAS, 2006-2009) at appraisal, specifically its second pillar to spread the benefits of economic growth to all, especially to those who were normally excluded. The specific approach of the CAS differed slightly in that it focused on Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) improving service delivery through local government and community-driven interventions, while the project aimed to strengthen NGOs in parallel with capacity building at the central government level. The Bank's Country Partnership Framework (CPF, 2016-2020) at closing and the government's seventh Five-Year Plan (2016-2020) both acknowledged the project's objectives and activities as important building blocks toward future inclusion. Incorporation of marginalized groups is one of the CPF's foundational priorities. The original objective promised outcomes that were broad, difficult to operationalize, and difficult to measure in the absence of rigorous evaluation (given the influence of non-project factors). The revised objectives were more readily monitorable, with outcomes more readily attributable to the project. Rating Revised Rating Substantial High b. Relevance of Design The original design involved a combination of support to government organizations, JPUF (semi- government), and NGOs. The bulk of this support ($18 million out of $35 million) was to be allocated for the provision of services, primarily through NGO subprojects. This implementation modality aimed at ensuring greater access and utilization, specified as outputs in the project's original objectives. In addition, the project was also to provide support to a UNICEF-administered program that targeted street children, which would contribute to utilization. The subprojects were to be awarded after ensuring that proposals met necessary quality standards. This model, which requires strong capacity among participating NGOs, has been applied in other countries and is based on sound public administration practice. JPUF and the government were to ensure that the NGOs had this necessary capacity. However, the assumption that JPUF and the government had the capacity to supervise NGOs was not warranted, given that the project also aimed to strengthen the government and JPUF from a weak basis; this represented a shortcoming in design. Most importantly, planned activities directly to strengthen equity and social inclusion were not clearly specified, which left specific gaps in the results chain connecting outputs with intermediate outcomes and final outcomes. Project design was revised significantly when the government's approach towards the provision of public services changed to a model that emphasized the direct government provision of public services rather than using NGOs. The new design reflected this change by supporting JPUF's new role as a government agency, particularly in supporting more uniform quality standards. The revised design also still allowed for contracting of NGOs as required, with the opening of more service centers at the upazila level and increasing the number of mobile service centers. Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) The revised design also strengthened the administrative capacity of MOSW, including in its role as service provider. These changes were reasonable to improve the quality of delivery and, to a lesser extent, increase access. Rating Revised Rating Modest Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To promote equity and social inclusion Rationale Outputs 3,045 government officials and service providers at union, upazila, and district levels were trained on disability issues, exceeding the target of 2,000. The original target had been set at "50% of government officials and service providers trained," and the original indicator was worded as "trained on the rights of persons with disabilities and available services." 546 government officials and service providers at union, upazila, and district levels were trained on child protection issues, not reaching the target of 2,500. The ICR (p. xiv) stated that the target was overestimated due to inclusion of staff not directly involved in service provision. Two spot surveys were conducted (in 2014 and 2016) to assess awareness on specific disability issues and need for relevant care, not reaching the target of 5 to 8 surveys. Outcomes The percentage of interviewed individuals aware of key government programs for persons with disabilities and children-at-risk increased from 12% (persons with disabilities) and 9% (children-at-risk) in 2011 to 29.4% in 2016, exceeding the target of 20%. The ICR (p. ix) was not clear on the source for the baseline data, and the baseline and results data were not precisely comparable across time, as the results appeared to group awareness of programs for persons with disabilities and children-at-risk into a single survey question. Indicators to measure the participation of persons with disabilities in the workforce or in self-employment, and school attendance among participating children, were dropped. Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) Overall, the results chain connecting project activities with increasing equity and social inclusion was not clear, and there were no indicators directly measuring these outcomes. Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 2 Objective [This objective field is intentionally left blank.] Rationale [See below for the next objective, which was added at the time of the first restructuring in June 2011.] Rating Not Rated/Not Applicable PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective Expand coverage of services for persons with disabilities and children at risk Revised Rationale This objective was introduced at the June 2011 restructuring. Outputs The project provided training to relevant government officials and personnel. The MOSW was, at the time of the ICR, reported to be taking the lead in formulating policies and programs consistent with its institutional mandate. JPUF staff were provided with training and study tours. The ICR did not provide information on the number of staff who benefited from these trainings and study tours or on their content. Guidelines and tools for disability certification were partially developed under the project. In 2013, this activity was dropped from the project and instead continued with government funding. The ICR (p. xiii) reported that the work was expected to be completed by the end of calendar year 2016. Institutional guidelines for child protection were created. At project closure, JPUF had service centers in all 64 districts of the country, meeting the target. 50 DSCs were established and made functional, meeting the target, and the other 53 DSCs in the country (created by Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) the government outside the project) were provided with equipment upgrades and staff training. 32 mobile vans were supported. Overall, 2.4 million service transactions were carried out at the DSCs, far exceeding the target of 0.5 million. 11 ICPSCs were established and made functional in seven divisional cities, exceeding the target of 7. Indicators measuring the percentage increase in the number of NGOs, DPOs, and community groups capable of developing and implementing rehabilitation and other disability programs as well as specialized services were dropped at the 2011 restructuring. Outcomes The number of disabled persons receiving services from JPUF increased from 35,000 in 2008 to 370,464 in 2016 (209,939 male and 160,525 female), exceeding the original target of a 10% increase. At the 2013 restructuring, the baseline was revised to 8,168 based on the availability of digitized records, and the target was set at 100,000; the achievement of 370,464 exceeds this revised target. 23.1% of persons with disabilities were receiving services via mobile outreach at project closure, exceeding the target of 10%. At project closure, 6,193 children-at-risk (3,017 male and 3,176 female) were registered with ICPSCs, presumably exceeding the original target of "double baseline" (baseline data have never become available), and doubling the revised target of 3,000. It should be noted that the wording of the indicator was changed from "children receiving child protection services" to "children registered with ICPSCs," which limits comparability. In 2008, none of 8,080 known street children were attending formal or non-formal education or training. According to DSS data, 500 of 2,100 known, not-reintegrated children (24%) were receiving technical/vocational education in 2016, not reaching the original target of 30%. Assistive devices were distributed to 3.48% of registered clients, not reaching the target of 10%. The ICR (p. xv) explained that this result was based on the 2014 and 2016 spot checks; the latest data based on actual device distribution and procurement, from FY 2014, indicated 11.4% coverage. The project had 376,657 direct beneficiaries, exceeding the target of 103,000. 43.4% were female, exceeding the target of 40%. An indicator to measure early detection of disabilities and impairments was dropped. Revised Rating Substantial PHEFFICACYTBL Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) Objective 3 Objective [This objective field is intentionally left blank.] Rationale [See below for the next objective, which was added at the time of the first restructuring in June 2011.] Rating Not Rated/Not Applicable PHREVDELTBL PHINNERREVISEDTBL Objective 3 Revision 1 Revised Objective Improve quality of services for persons with disabilities and children at risk Revised Rationale This objective was introduced at the June 2011 restructuring. Outputs In addition to the outputs reported under Objective 2: Workshops and reports on the performance of disability and children-at-risk programs were held/prepared in 2014 and 2016. Minimum standards of care and codes of conduct for working with street children were developed and shared with ICPSC staff. No information was provided in the ICR on the application of those standards (which was an original target that was dropped). Standards of care for children-at-risk were developed, meeting the target. An indicator for development and piloting of an alternative care model for orphans was dropped. JPUF, DSS, DSCs and ICPSCs developed fully operational MISs, and more than 800 people were trained in their use. All client records were digitized and regular reports produced. Outcomes The DSCs were reported to provide a "rich number of services" (ICR, p. 33), including counseling, testing, physiotherapy, device distribution (wheelchairs, standing frames, crutches, artificial limbs, hearing aids, etc.), and group therapy play and a toy library for children with disabilities. The ICPSCs were reported to provide gender-segregated shelter, food, clothing, medical facilities with psychosocial protection, and education. Each ICPSC was staffed by 20-24 appropriately trained personnel, including two psychologists. Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) 93.53% of beneficiaries reported satisfaction with DSC services, equipment, and staff behavior/competence, surpassing the original target of 80% and the revised target of 70%. It should be noted that the wording of the indicator was changed from being "fully satisfied" to "reporting satisfaction" at the 2013 restructuring. 90% of children reported satisfaction with services at ICPSCs in 2016. The ICR (p. 13) reported these satisfaction data based on third-party spot checks conducted in 2014 and 2016 that assessed the functionality of service centers, staff skills, staff satisfaction and motivation, and beneficiary satisfaction. No specific information was provided on the survey population or sampling parameters. The share of DSC staff achieving a minimum score on a technical skills assessment decreased from 66% in 2013 to 60% in 2016, not reaching the target of 80%. The ICR (p. x) noted that these results were based on aggregation and harmonization of various certificates and accreditations, and therefore the results were not directly comparable to baseline. The ICR (p. 25) reported that, according to the spot checks conducted in 2014 and 2016, DSCs showed high levels of functionality. Over 80% of centers in both rounds of checks had necessary infrastructure and facilities, and both scored around 80% on an overall equipment index (measuring daily cleaning, existence of a reception desk, and internet, electricity, and tap water connections). However, only 50% had an accessibility ramp, primarily because of the use of rented premises. The ICPSCs showed "remarkable improvement" between 2014 and 2016 (exact data were not provided), with 100% of them functioning with basic infrastructure and facilities at the 2016 check. The percentage of staff with basic knowledge of disability laws and rules increased at the DSCs from 18% in 2014 to 100% in 2016, and at the ICPSCs from 46% in 2014 to 98% in 2016. The percentage of ICPSC staff who had knowledge about the role of ICPSCs for children-at-risk increased from 61% in 2014 to 100% in 2016. Overall staff competency was reported to have increased from 65% in 2014 to 89.7% in 2016; it is not clear how this was measured (ICR, p. 25). 70% of children receiving services at ICPSCs were reintegrated, exceeding the target of 25%. Almost 25% of children who were not reintegrated received training that led to their employment or self-employment. Based on information reported in the ICR, most outcome targets were met or exceeded. Issues noted above in some baseline data, consistency of definition of indicators, and measurement of indicators, however, are considered indicative of weaknesses in the evidence underpinning the reported results. Achievement of this objective is therefore considered barely Substantial. Revised Rating Substantial PHREVISEDTBL 5. Efficiency The PAD (pp. 70-78) found the annual cost of disability in the country to be US$ 1.2 billion, or 1.74% of gross domestic product, due to reduced employment among persons with disabilities; foregone schooling among Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) disabled children; foregone income of adult helpers; and foregone schooling of child helpers. This amounted to US$ 148 annually per disabled person. The annual loss in lifetime earnings of street children foregoing schooling was estimated at US$ 9 million. The ICR (p. 35) stated that a full economic and financial analysis following the PAD's methodology was not possible at the ICR stage, as the project could not claim credit for outcomes related to education or employment. The ICR explained that the output-based nature of the project and lack of impact evaluation limited the ability to assess gains and therefore conduct a meaningful quantitative assessment. A cost per beneficiary of US$ 35.85 for the DSCs and US$ 1,203 for the ICPSCs was calculated. There was evidence of significant shortcomings in implementation efficiency. The early 2009 change of government required a complete overhaul of the approach to the disability services component, including a restructuring of JPUF. The ICR (p. 11) stated that "both of these processes were time taking." Furthermore, the composition of the Bank's project team changed almost completely during early implementation, requiring the establishment of "almost a new dialogue" and incurring additional delays (ICR, p. 11). The June 2011 restructuring discussions took about 18 months, and that restructuring was not declared effective until April 2012. A slow implementation pace was also noted at the 2013 and 2015 restructurings due to low capacity of the implementing agencies and cumbersome government approval procedures (ICR, p. 12). In addition, shortcomings in financial management and misprocurements likely hindered efficiency (see Section 11b). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the original objective, relevance of objectives is rated Substantial, relevance of design is rated Modest, achievement of the single objective is rated Negligible, and efficiency is rated Modest, producing an Outcome rating of Unsatisfactory. Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) Under the revised objectives, relevance of objectives is rated High, relevance of design is rated Substantial, achievement of both objectives is rated Substantial, and efficiency is rated Modest, producing an Outcome rating of Moderately Satisfactory. Under OPCS/IEG harmonized guidelines, the overall Outcome rating is calculated according to the percentage of disbursements under each set of objectives. In this case, 10.6% of Bank financing was disbursed under the original objective. Original objectives: Unsatisfactory (2) * .106 = 0.21 Revised objectives: Moderately Satisfactory (4) * .894 = 3.58 0.21 + 3.58 = 3.79, rounds to 4 = Moderately Satisfactory. The project's overall Outcome is rated Moderately Satisfactory, consistent with moderate shortcomings in preparation and implementation. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating The Ministry of Finance and MOSW have allocated funding to continue support for the service centers established or strengthened under the project, including continuation of full staffing, but the ICR (p. 26) expressed uncertainty about budgetary commitments over the long run. The project team later added that the project's interventions for children-at-risk are slated for rollout to 19 additional districts using government resources. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry Preparation, which took less than ten months after the government's request for a project, included consultation with stakeholders (including the National Forum for Organizations Working with the Disabled, UNICEF, and a range of childrens NGOs) and a Quality Enhancement Review. The component covering children-at-risk built on an existing UNICEF project model. However, the disability services component was a first project of its type for the World Bank and therefore had no specific best practices or documented Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) lessons to follow. Some key general lessons were learned and reflected in project design (PAD, pp. 8-9): the importance of having a national disability organization with independence, transparency/accountability, and professional and transparent financial management and procurement systems (these were conditions for World Bank support, and were considered to be achieved through the restructuring of the former National Foundation for the Development of Disabled Persons into JPUF); the need for relatively simple institutional and reporting arrangements; the need to take into account time requirements for internal government processes and procedures; and the imperative of training project entities on Bank fiduciary guidelines prior to appraisal. The PAD also discussed the importance of selecting only a few relevant, realistic, measurable outcome indicators, but this lesson was not well reflected in project design. Some risks were identified as Substantial at appraisal, even after mitigation: that coordination among ministries would be weak (addressed by naming secretary-level officials and independent figures of high reputation to the JPUF Board); and that training of MOSW staff would not result in long-term capacity building (addressed through ongoing institutional assessment and training). However, there were some significant shortcomings. The project's planned activities were complex in relation to existing capacity of NGOs/DPOs in the disability sector, a risk that was identified in the PAD but insufficiently mitigated, as mitigation measures provided for capacity building for JPUF but not for the NGOs/DPOs themselves. The ICR (p. 26) also suggested that additional capacity enhancement measures for MOSW, JPUF, and DSS would have been prudent, including hiring of key fiduciary staff as a condition of project readiness. The PAD's risk assessment also lacked a political economy analysis, despite the fact that the December 2008 elections were already scheduled. Lack of attention to these issues led to "almost absent progress" prior to the 2011 restructuring (ICR, p. 26); the ICR (p. 30) noted that the initial implementation challenges resulting from inadequate preparation "posed reputation risks" for the Bank in Bangladesh's social protection sector. In addition, the initial results framework had significant shortcomings (see Section 10a). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The task team leader (TTL) and most of the project team changed soon after the project became effective. The new team nevertheless worked to transition the dialogue with the new government to find ground for consensus. Given the need to build trust and the complexity of the project, it took 18 months for restructuring-related discussions to conclude. After the first restructuring, the Bank team provided numerous opportunities for sharing of international best practice and relevant staff training. The project's restructurings responded to a change in government priority and to learning from implementation experience, dropping less successful and relevant activities at the second and third restructurings, and taking advantage of cost savings to cancel some project funds. The project team later highlighted the provision of extensive support and training to the implementing agencies following several misprocurements in 2011 and 2013 (Section 11b). The ICR (p. 28) cited the "impressive hard work of the project team" in improving disbursement, implementation, and achievement following the first restructuring. Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Government commitment to the project's objectives and activities remained strong, even after the December 2008 elections, when the new government shifted the official stance on service delivery. With the advent of the new government in 2009, MOSW worked closely with other line ministries and the Bank team to revise the project in light of new priorities. After the 2011 restructuring, the government took key decisions to create additional service centers with cost savings. 53 DSCs were established independent of the project, with government financing. However, there were moderate shortcomings. Cumbersome approval and clearance procedures significantly slowed the pace of implementation at several key points, including a delay of almost a year between Board approval of the first restructuring (June 2011) and its effectiveness (April 2012). The ICR noted, however, that the speed of government clearances improved significantly during the project's final three years (p. 28). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The disability component was to be implemented by the newly restructured JPUF, and the children-at- risk and institutional development components by DSS (MOSW) through the PCAR unit. The DSS Director General was to serve as Project Director. As already noted, JPUF underwent a significant transformation early in the project period. Both agencies implemented project activities effectively following the first restructuring. However, the ICR (p. 12) noted that the low capacity of implementing agencies slowed progress, especially on procurement and financial management (Section 10b). Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) 10. M&E Design, Implementation, & Utilization a. M&E Design The original results framework had four outcome indicators, three expressed in terms of percentage increase from baseline. Baselines were specified in the PAD (p. 28). One indicator measured coverage of services for disabled persons, but there was no clear measure for quality of social care services or for equity and inclusion. Indicators for increased participation of persons with disabilities in the workforce and of street children in education/training may have been conceptualized as related to equity and inclusion, but their results could not reasonably have been tied to project-financed activities. As the ICR stated (p. 12), even if the project had not been restructured, the M&E framework still would have required revisions in order to track progress adequately. Regarding completion of activities, achievement of outputs, and achievement of those outcomes that were reasonably well defined, M&E arrangements were well specified in the PAD. The JPUF was to have a dedicated M&E department responsible for monitoring outcomes related to disability. Outcomes related to children-at-risk were to be monitored primarily through the PCAR project unit, while M&E systems of the DSS were under development. For monitoring of institutional development activities, DSS personnel were to be trained in M&E methods as well as policy and program development. JPUF, DSS, and NGO staff were to be trained in data processing (storage), analysis, and reporting, and community groups were to be trained to participate in the monitoring exercise to be able to assess the situation of their communities and families. b. M&E Implementation Outcome indicators were appropriately revised at the 2011 restructuring to measure both coverage and quality of services; however, the revisions for some indicators introduced issues of comparability between baseline and endline.. Relevant outcome indicators were maintained through the 2013 and 2015 restructurings. Some output indicators that measured activities no longer included in the project were dropped. The results framework from 2011 onward included intermediate results indicators that expressed the project's logic connecting activities and desired outcomes. During implementation, all indicators were carefully monitored using both administrative data and spot checks. Two third-party spot checks covered DSCs, ICPSCs, beneficiaries, and communities. c. M&E Utilization According to the ICR (p. 13), M&E findings were used by the implementing agencies to identify changes necessary to improve quality of services. Stakeholder workshops in 2014 and 2016 included sharing of the results of spot checks and process studies, leading to collaborative identification of approaches to improve performance. The survey instruments used during the spot checks were customized for JPUF and DSS for the purposes of future monitoring activities. Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) M&E Quality Rating Modest 11. Other Issues a. Safeguards Based on an environmental screening done in consultation with the regional safeguards team, the project was rated Environmental Assessment category C (PAD, p. 79). No safeguard policies were triggered. The ICR (p. 15) noted that project document referrals to DSCs or ICPSCs as having been "built" or "established" did not mean actual construction, and that all services were provided through rental of existing premises. b. Fiduciary Compliance Financial management. Both DSS and JPUF had staffing issues related to financial management, but according to the ICR (p. 14), compliance and reporting were appropriate. The Bank team raised occasional minor concerns over incomplete documentation and accuracy/timeliness of withdrawal applications. Lack of capacity affected DSS financial statements, which received qualified opinions from auditors for FY 2012 and FY 2013. Shortcomings of both agencies in financial planning resulted in reduced budget allocations from MOSW in the FY 2013-2014 budget. A month before closing, JPUF experienced funds shortages as a result of national payscale increases, underforecasting of expenditures, and underbudgeting for XDR depreciation. According to the project team, the implementing agency took measures to streamline bookkeeping, and the task team actively followed up and provided technical assistance. At closing, there were no outstanding interim financial reports or audit reports. The project team confirmed that the project was overall in compliance with the Bank's fiduciary policies. Procurement. DSS and JPUF were responsible for all procurement under the project. Technical assistance was provided to both agencies, and a separate procurement unit was set up under JPUF. By the time of the 2011 restructuring, both DSS and JPUF had hired procurement specialists; however, they were unable to retain qualified staff due to below-market salaries. As a result, weak capacity remained a challenge throughout the project period. All planned procurement was completed during the implementation period. Four cases of misprocurement were identified between FY 2011 and FY 2013, covering two contracts from DSS and two from JPUF that totaled US$ 50,000. As of August 2016, the ineligible expenses were in the process of being refunded (ICR, p. 13); the project team confirmed that these expenditures have been fully refunded. c. Unintended impacts (Positive or Negative) None reported. Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Risk to Development Modest Modest --- Outcome Moderately Moderately Bank Performance --- Satisfactory Satisfactory Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR (pp. 30-31) offered insightful lessons emerging from the project's experience, including: Investment of adequate time and resources in quality at entry facilitates smooth implementation, especially for projects that are first-of-a-kind, or the first project in a portfolio of a given global practice. In this case, insufficient attention to capacity issues of the implementing agency and political economy analysis led to slow implementation in the project's early years and ultimately the need for significant restructuring. Complexity of project design is most prudently matched with the capacity of the implementing agency. In this case, initial design was not tailored to the context of a very complex challenge in combination with very low- capacity implementing agencies. Subsequent restructurings were all an exercise in simplification and streamlining of the design (ICR, p. 30). Investments in political economy analysis can pay off. In this case, with an election upcoming just after appraisal, it is possible that signs of changes in the government's stance on social welfare policy could have been anticipated (ICR, p. 30) and taken into account in project design. 14. Assessment Recommended? Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Disability and Children at Risk (P106332) Yes Please explain This project was among the first World Bank-financed projects to focus on issues of disability. Lessons on what worked would be valuable for future operations in social protection and other sectors. These lessons could also prove useful for mainstream projects that aim to address disabilities in their design. 15. Comments on Quality of ICR The ICR was candid and well argued, marshaling evidence from a variety of sources to assess achievement of objectives. It laid out the institutional development of key agencies clearly and concisely, and it explained thoroughly the rationale for the project's restructurings. The efficiency analysis, as well as the discussion throughout, provided key information on implementation challenges and procedures employed to overcome those challenges. Some information later shared by the project team could have been usefully included in the ICR. The discussion of safeguard policy compliance was careful and thorough, though more information and analysis could have been provided on risks to development outcome. In addition, more attention could have been paid to the comparability challenges posed by revisions in the definition of some indicators. a. Quality of ICR Rating Substantial Page 20 of 20