Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) Report Number: ICRR0021740 1. Project Data Project ID Project Name P123933 CCRES Country Practice Area(Lead) Philippines Environment, Natural Resources & the Blue Economy L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-15409 31-Dec-2018 4,500,000.00 Bank Approval Date Closing Date (Actual) 30-Sep-2013 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 4,500,000.00 4,500,000.00 Revised Commitment 4,500,000.00 4,500,000.00 Actual 4,500,000.00 4,500,000.00 Prepared by Reviewed by ICR Review Coordinator Group John Redwood Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to both the Global Environment Facility (GEF) grant agreement (Schedule 1, pg. 4) and the Project Paper para. 12, pg. 11), the project development objective (PDO) was "to design and support the uptake of innovative models for valuing mangrove, seagrass and coral reef ecosystem services with the potential to enhance the sustainability of marine-based enterprise and marine spatial planning in select coastal communities in Indonesia and the Philippines." Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) For purposes of the assessment of project Efficacy, the ICR divided this PDO into two objectives: (i) to design models for valuing mangrove seagrass and coral reef ecosystem services with the potential to enhance the sustainability of marine-based enterprise and marine spatial planning in select coastal communities in Indonesia and the Philippines; and (ii) to support the uptake of innovative models for valuing mangrove seagrass and coral reef ecosystem services with the potential to enhance the sustainability of marine-based enterprise and marine spatial planning in select coastal communities in Indonesia and the Philippines. This evaluation will adopt the same approach. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1. Quantifying the value and market potential of coral reef and mangrove ecosystem services. (Appraisal Cost: US$ 2.4 million; Actual Cost: US$ 2.2 million). This component aimed to demonstrate how ecosystem services can be accurately valued and systematically managed to deliver pro- poor, pro-environment outcomes and to help build the political rationale for change. Working in a number of field sites in Indonesia and the Philippines, it would first identify the value of key coral, mangrove and seagrass ecosystem services as a function of system state. These services would include reef fisheries, ornamental species, island stability, prestige diving tourism, blue carbon, and cultural benefits. It would also help operationalize the use of ecosystem services for marine spatial planning. Tools would be developed and customized to allow stakeholders to visualize the production and flows of ecosystem services and therefore evaluate the consequences of different scenarios for development and management. Component 2. Forging community-led innovation in capturing and sustaining benefits from marine ecosystem services and enhancing resilience in the face of climate change. (Appraisal Cost: US$ 2.3 million; Actual Cost: US$ 1.9 million). This component was expected to empower communities to move away from unsustainable coastal resource use practices and towards activities that support improved ecosystem health, increased production outcomes and greater resilience as the result of livelihood diversification through the development of sustainable alternative enterprises and new income generating opportunities. Drawing on the results of Component 1, it would propose and develop customized business models that, once implemented, could support the health of marine and coastal ecosystem services. To do this working closely with local communities and integrating with Components 1 and 3 at key points, it would scope the local and external business environments, identify and assess the individual and interactive performance of potential eco-business models that are able to support and enhance ecosystem service values. Component 3. Promoting behavioral change through outreach, decision support and regional learning from results at field sites. (Appraisal Cost: US$ 1.4 million; Actual Cost: US$ 1.3 million) This component would work with key stakeholders on the ground to share project products and findings and help channel these into appropriate policy and management outlets. Focusing on the uptake of knowledge generated from Components 1 and 2, it would develop a series of strategies and activities designed to: (i) facilitate the uptake of the valuation and enterprise models into policy, management and future project design; and (ii) increase the awareness and understanding of communities at the field sites of the linkages Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) between the services the coastal and marine ecosystems provide and their livelihoods and health. This would be undertaken through reciprocal engagement with stakeholders, placing strong emphasis on local partnerships, local leadership, and culturally responsive approaches to gathering information about the desires, attitudes, and current behaviors of individuals in demonstration sites. Component 4. Project coordination and management. (Appraisal Cost: US$ 0.4 million; Actual Cost: 0.7 million). This component would be implemented by establishment of a dedicated Project Coordination Unit (PCU) in the Global Change Institute at the University of Queensland, which would serve as the Project Executing Agency (PEA), to oversee project implementation, monitoring and evaluation (M&E), outreach and communication activities, and future planning, including development activities to identify future do- financing and new partnerships. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Actual project cost according to the ICR (Annex 3, pg. 50) was US$ 6.044 million, or 93 percent of the appraisal estimate of US$ 6.5 million. Financing: The project was financed by a US$ 4.5 million GEF grant, of which 91.5 percent was disbursed according to the ICR. Grantee Contribution: The contribution by the grantee (University of Queensland) was estimated at appraisal to be US$ 2.0 million and the actual contribution, according to the ICR, represented 96.3 percent of this amount (US$ 1.927 million). Dates: The project was approved on September 30, 2013, became effective on November 6, 2013 and was closed, as planned, on December 31, 2018. 3. Relevance of Objectives Rationale Project objectives were and remain of high relevance. The project region's coastal and marine ecosystems - - mangroves, coral reefs, and sea grass meadows -- are of global biological and national and local economic importance. They are the first line of defense against storm surges and sea level rise for low lying coastal areas and by sequestering large amounts of carbon, thereby helping to mitigate climate change. Local fisheries also support livelihoods and food security for large poor populations residing in coastal communities. Project objectives were also well aligned with the contemporaneous and current World Bank Group Country Partnership Strategies (CPSs) and Frameworks (CPFs) for both nations. As concerns the CPF for Indonesia for FY 16-20, they are in sync with two key engagement areas: (i) to ensure holistic natural resources management and (ii) to expand the maritime economy and build sustainable livelihoods for the poor. Regarding that for the Philippines for FY15-18, they align with engagement area No. 4, increasing resilience and improving natural resource management and sustainable development. They likewise reflect development priorities identified in the respective national development plans, covering the period 2005-2025 in the case of Indonesia and 2011-2016 in that of the Philippines. According to the ICR (para. 15, pp. 9-10), moreover, they were likewise fully consistent with a shared objective by these two Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) countries with respect to sustainable natural resource management around the marine and coastal environment as reflected in their joint engagement in the Philippines-based Partnerships in Environmental Management for the Seas of East Asia (PEMSEA), which "seeks integrated solutions for effective management of coastal and marine areas to have positive impact on communities through enhanced food security and livelihood opportunities, amongst other focal areas," as well as in their joint commitment to the "Blueprint for the ASEAN Socio-Cultural Community 2025," which is the guiding mandate for the ASEAN (Association of Southeast Asian Nations) Working Group on Coastal and Marine Environment (AWGCME). Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To design innovative models for valuing mangrove, seagrass and coral reef ecosystem services with the potential to enhance the sustainability of marine-based enterprise and marine spatial planning in selected coastal communities in Indonesia and the Philippines. Rationale Theory of change: The desired outcome in relation to this objective was improvement of the knowledge and information base necessary to enhance the sustainability of marine-based enterprise and marine spatial planning in selected coastal communities in Indonesia and the Philippines. The activities (inputs) that would lead to this outcome were the collection of data to underpin models and quantify the value of ecosystem services, scoping and designing the pertinent models, and training and building local community awareness and capacity to participate in this process. The corresponding outputs were the development of innovative models and tools for marine spatial planning, marine-based enterprise, and ecobusiness development, systems analysis, and behavior change with respect to the value of coastal and marine ecosystems, which is a necessary precondition for the adoption of better decision making and enhancing their value, including on the part -- and to the benefit of -- the local communities involved. The underlying assumption in this regard was that training and capacity building efforts would result in the required behavioral change and local willingness to apply the knowledge generated under the project. This results change and underlying assumption were logical and appropriate. Outputs:  4 models of ecosystem function and services valuation developed compared with an appraisal target of 3 (125 percent achieved), specifically: Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) o Coastal Protection tool, which assesses coral reef ability to protect shorelines and enables planners to determine which reefs best protect key coastal infrastructure and communities; this tool also provides information on how wave conditions and wave forces on corals will change with sea level rise or loss of reef elevation, and how the shoreline may recent behind fringing reefs; o Reef React (model) to help guide policy and management interventions to reduce negative impacts of climate change on human activities, this model assists users to predict alternative futures for coral reef ecosystems under various climate and human use scenarios; o Food web model, whose results were distributed as a policy brief ("Priority Reefs for Conservation and Fisheries Replenishment") containing guidelines for prioritizing which reefs are best suited to marine biodiversity conservation compared with those best used for reef fisheries; and o MPA (Marine Protected Area) toolkit, consisting of five elements: (a) MPA policy brief ("Healthy Fisheries through Marine Resources"); (b) MPA Placement Optimization Tool; (c) MPA Size Optimization Tool; (d) Fish SPACE (Fisheries for Sustaining Peoples Access through Conservation and Equitable Systems) -- a spatial planning tool that highlights the consequences of alternative decisions on total MPA coverage, placement, and local size; and (d) Educational Tool for Marine Design. When used together, according to the ICR (pg. 37), these elements provide "an innovative and comprehensive support system for marine reserve design."  5 decision support systems for marine reserve design to optimize management objectives developed compared with an original target of 2 (250 percent achieved) - this refers to the five parts of the MPA toolkit referred to above.  A policy brief to communicate the value of seagrass beds as planned.  Development and analysis of 5 system maps for each site detailing interactions between specific ecosystem services and socio-economic sectors, compared with a target of 4 (125 percent achieved) - - four maps were developed for the Philippines (for food security, water pollution, fish catch decline, and mangrove loss) and one for Indonesia (reef fisheries decline) through focus group discussions with stakeholders in El Nido and Selatar, respectively, then combined by project teams into one systems map for each problem.  2 business toolkits, the Ecosystem-based Business Development (EbBD) model (which uses ecosystem services and biodiversity as part of an overall sustainable development strategy to help support sustainable livelihoods and local economic development in low resource coastal communities) and the ECO-BIZ Program (which seeks to identify local entrepreneurs and to assist them in establishing or expanding businesses that supported the local economy in a way compatible with local ecosystems or that leveraged local ecosystems in a sustainable way) were developed, exactly meeting the appraisal target (100 percent achieved).  3 Systems Analysis toolkits as planned - (i) SESAMME (Socio-Ecological Systems App for Mental Model Elicitation), which captures information, including past and future trends and the current state of system components (such as resources, activities, pressures, and decisions) from local communities and helps them to visualize how these components interact; (ii) SYSTORY, which assists managers to understand and visualize the dynamics of coastal systems and assess the influence of alternative scenarios on system trajectories over time; and (iii) the Systems Simulation Tool, which quantifies interactions between activities on land (such as farming and urban development), activities on water (such as fishing), coastal ecosystems such as coral reefs and mangroves) and coastal resources (such as fish) that allow the user to simulate the behavior of a coastal system over time -- were developed. Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933)  2 Behavior Change toolkits -- (i) My Future, My Oceans, which is a low-cost process of behavioral diagnosis and capacity enhancing that seeks to empower individuals in coastal villages to adopt behaviors that lead to healthier families, happier lives, and a cleaner environment; and (ii) FishCollab, which is intended to assist governments, communities, and NGOs to work together to improve coastal management by enabling users to identify key stakeholders, develop networks (i.e., find and develop cooperation with relevant and committed parties), analyze policy, analyze and reduce conflict, and identify opportunities and challenges using local knowledge and science. Outcomes:  9 planning frameworks were designed that incorporate models of ecosystem functions and services, including coastal defense and decision support systems, developed in response to stakeholder demand applied or appear likely to be applied, compared with a target of 2 (450 percent achieved). These include; o Application of the MPA toolkit in 27 locations across Indonesia across 34 Indonesian provinces by WWF (World Wildlife Fund) and MMAF to support decision-making in response to demand from stakeholders; o RARE, a NGO, plans to apply marine planning tools in Belize, Honduras, and Mexico in the Caribbean; o Haribon Foundation, partner of Department of Natural Resources (Philippines)/UNDP (United Nations Development Programme) Smart Seas PH (Philippines) Project has used FishSPACE to evaluate marine reserve design and fisheries management initiatives in Lanuza Bay, Surigao del Sur; o Smart Seas PH is using Lanuza Bay as a model site to roll out the MPA design tools in other Marine Key Biodiversity Areas such as the Davao Gulf and the Verde Island Passage. o The Palawan Council for Sustainable Development and WWF (World Wildlife Fund) Palawan are using FishSPACE and the MPA Size and Placement Optimization tools to support MPA design in Cluster 5 municipalities in northeastern Palawan, which includes El Nido, Linapacan, Taytay, Dumaran, Roxas, and Araceli; o FishSPACE and MPA Size and Placement Optimization tools were used in Oriental Mindoro, Lanuza Bay, Palawan, and Batangas; o Healthy Fisheries through Marine Reserve policy brief and MPA Placement and Size Optimization tools were applied by the local government and NGOs to plan MPAs in three sites in Indonesia (Seram, Sulawesi Tengarra, and Simeulue, Aceh) and one in the Philippines (Tanon Strait Protected Seascape); o The Zoological Society of London, working in the Panay and Negros islands, and the Macajalar Bay Development Alliance in Misamis Oriental have developed work plans to use the MPA tools; and o The Management Board of the EL Nido-Taytay Resource Protected Areas in the Philippines will consider using FishSPACE in the planning of protected areas management.  The EbBD model and ECO-BIZ Program were both piloted at the Selayar (Indonesia) and El Nido (Philippines) sites with positive results. In terms of uptake, the Selayar community helped to forge businesses directly as a result of the EbBD engagement at the site, including due to the collaboration of several villages (Patikarya, Barat Lembongan, Gusung Barat, and Bahuluang) to implement community-based tourism through homestays and the Waste2Enterprise (W2E) initiative that empowered villagers to use business solutions and community-based management to address ocean plastics. In El Nido, the ECO-BIZ Challenge generated responses from 56 applicants, half of whom Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) were selected for seed funding and hands-on training in basic planning, accounting, and marketing skills for entrepreneurship, which helped to result in community-led innovation projects for: (i) using coconuts as an alternative to mangrove wood; (ii) ecotourism in mangrove areas; (iii) cultivating giant bamboo to reduce logging in native forests; and (iv) supplying ornamental native flowers to the tourism industry. In Selayar, the competition attracted 143 applications with 53 finalists for training and resulted in three start-up ideas that received seed financing from the project: (i) an aquaculture project focusing on floating grouper and lobster grow-out cages; (ii) an educational tourism approach focused on mangrove conservation; and (iii) a handicrafts shop which makes products from recycled waste. As a small Technical Assistance (TA) project implemented by a University research center, its outcomes in relation this objective are, in fact, more in the nature of consolidated outputs. It can thus be concluded that the project's Efficacy in relation to this objective was Substantial. Rating Substantial OBJECTIVE 2 Objective To support the uptake of innovative models for valuing mangroves, seagrass and coral reef ecosystem services with the potential to enhance the sustainability of marine-based enterprise and marine spatial planning in selected coastal communities in Indonesia and the Philippines. Rationale Theory of change: The desired outcome in relation to this objective was the delivery of support for the uptake of innovative models for valuing mangroves, seagrass, and coral reef ecosystem services. The activities (inputs) undertaken by the project to achieve this outcome were proactive dissemination, outreach, and learning activities at the local, national, regional and global levels through training, capacity building and awareness raising to promote behavioral change and marketing, outreach, and dissemination of project outputs to promote uptake. These activities were considered necessary to encourage stakeholders to utilize the improved knowledge and information tools (outputs) in order to enhance the sustainability of marine- based enterprises and spatial planning in coastal communities in Indonesia, the Philippines, and elsewhere. The underlying assumptions in this regard were that: (i) adequate capacity would be built in provincial and community organizations to use the suite of models and other tools developed under the project; and (ii) there would be adequate uptake of the models and other tools developed under the project at the local, national, and regional levels to insure their institutionalization. This results chain and the underlying assumptions were logical and appropriate. Outputs:  Information and knowledge products were shared with local communities, governments, technical networks, NGOs, existing projects and the Strategic Partnership for East Asian Seas through 50 outreach, information sharing, and dissemination events, compared with an appraisal target of 40 (125 percent achievement) at pilot sites, nationally in Indonesia and the Philippines, regionally in East Asia, Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) and globally at marine conservation conferences. In 2018, for example, there were 10 such events (project activities), specifically: o Tools training workshop at Makasar, Indonesia (February); o Tools training workshop at Tagatay, Philippines (April); o The Third Targeted Regional Workshop for GEF International Waters (IW) projects from East Europe and Asia-Pacific (April-May) and the First Annual Asia-Pacific Regional Network Meeting (May); o The Asia-Pacific Coral Reef Symposium at Cebu, Philippines (June); o Site exit/closing visit for Selayar, Indonesia (July); o Our Ocean Conference, Bali, Indonesia (October); o International Waters Conference (November); o Site exit/closing visit for El Nido, Philippines (November); o PEMSEA's EAS Conference (November); and o Capturing Coral Reef and Related Ecosystem Services (CCRES) (November); and o CCRES eNews -- 4 editions (March, June, September, and December)  19 models and knowledge products to inform the design of regional and national projects, development plans or policies, and community-based coastal resource management were developed compared with an appraisal target of 5 (380 percent achievement).  18 plans or products to reduce stress and maintain value of ecosystem services were developed compared with an appraisal estimate of 5 (360 percent achievement) Outcomes:  19 models and knowledge products have been used or are expected to be utilized by projects/plans and 18 such outputs have been used or are expected to be utilized to reduce stress and maintain the value of ecosystem services, which in many cases are the same as identified in connection with Objective 1 above - specifically: o Largely at the request of the Ministry of Marine Affairs and Fisheries (MMAF), WWF Indonesia has applied the Marine Spatial Planning (MSP) toolkit in 17 locations/projects. o The COREMAP-CTI (Coral Reef Rehabilitation and Management Project -- Coral Triangle Initiative) has requested and received a training workshop of the use of tools by CCRES and the Reef React tools on resilience has informed the collaborative venture led by COREMAP in partnership with CCRES to develop the National Coral Reef Health Index, which has been applied to all COREMAP monitoring data on coral reefs. o The Government of Palau was planning to use the MPA toolkit in September 2018. o The Palawan Council for Sustainable Development (PCSD) in the Philippines plans to use the MPA toolkit to review the contribution of existing MPAs to fisheries in late 2018. o Selayar local community has applied BsBD and MFMO (?) to address waste management in the town. o Parak village community in Selayar has applied FishCollab to prepare community-designed Marine Protected Areas and management plan and submitted it for recognition and support by the provincial government. o The founder of Bali Seafood International (BSI) has returned to Selayar to commence a more detailed assessment of the site for its second Fisheries Management Center(FMC) there, following the successful launch of its first FMC in Sumbawa. o RARE intends to apply marine planning tools in Belize, Honduras, and Mexico in the Caribbean. A subsequent communication to IEG from the ICR authio confirmed that the tools Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) developed under the project have been applied in teh Caribbean context in order to assess priority reefs for conservation in Central America and the Caribbean. o The Gulf and Caribbean Fisheries Institute (GCFI) and the US National Oceanic and Atmospheric Administration (NOAA) organized an MPAConnect training workshop in Saba for MPA managers pf 10 Caribbean countries using the MPA Size Optimization tool. o Haribon Foundation, partner of Department of Natural Resources (Philippines)/UNDP (United Nations Development Programme) Smart Seas PH (Philippines) Project has used FishSPACE to evaluate marine reserve design and fisheries management initiatives in Lanuza Bay, Surigao del Sur; o Smart Seas PH is using Lanuza Bay as a model site to roll out the MPA design tools in other Marine Key Biodiversity Areas such as the Davao Gulf and the Verde Island Passage. o The Palawan Council for Sustainable Development and WWF (World Wildlife Fund) Palawan are using FishSPACE and the MPA Size and Placement Optimization tools to support MPA design in Cluster 5 municipalities in northeastern Palawan, which includes El Nido, Linapacan, Taytay, Dumaran, Roxas, and Araceli; o FishSPACE and MPA Size and Placement Optimization tools were used in Oriental Mindoro, Lanuza Bay, Palawan, and Batangas; o Healthy Fisheries through Marine Reserve policy brief and MPA Placement and Size Optimization tools were applied by the local government and NGOs to plan MPAs in three sites in Indonesia (Seram, Sulawesi Tengarra, and Simeulue, Aceh) and one in the Philippines (Tanon Strait Protected Seascape); o The Zoological Society of London, working in the Panay and Negros islands, and the Macajalar Bay Development Alliance in Misamis Oriental have developed work plans to use the MPA tools; and o The Management Board of the EL Nido-Taytay Resource Protected Areas in the Philippines will consider using FishSPACE in the planning of protected areas management. o The Government of Indonesia us funding a Institute of Marine Science (LIPI) project applying FishCollab and EbBD tools in 5 locations across the country in 2019 to develop effective interventions for tackling poverty in fishing communities. o A proposal involving the uptake of SASAMME and SYSTORY has been prepared for a tender entitled "Develop National Ridge to Reef R2R," which was expected to close in January 2019. o A proposal to be submitted to USAID was being developed by Palawan State University (PSU) and the Business Team on Marine Waste Recycling involving the application of business and behavioral change tools. While, As noted above, as a small TA operation, these outcomes are similar in nature to outputs, it can nevertheless be concluded that the project was able to substantially support the uptake of the models and tools developed under the project. Rating Substantial OVERALL EFF TBL OBJ_TBL Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) OVERALL EFFICACY Rationale This TA project has met and in most cases exceeded all of its appraisal targets both for the PDO and Intermediate Results Indicators. In addition to the above, it also succeeded in significantly surpassing the appraisal target for direct beneficiaries (3,001 compared with 1,000) and also exceeded the target for the percentage of beneficiaries that were female (47 versus 30 percent). The ICR (pp. 34-35) attributes this performance to "broad scoping and community engagement exercises, especially in model development for tools related to: (i) behavior change; (ii) systems thinking which engaged project beneficiaries through forums such as community workshops and focus group discussions. In a subsequent communication with IEG, the ICR author clarified that: (i) the direct beneficiaries consisted primarily of the local community and government members at the two pilot sites that participated in the information gathering and modeling activities and were trained under the project and (ii) the actual number greatly exceeded the original estimate because at the time of appraisal the extent of the data collecting activities and training requirements was not fully known and, in practice, turned out to be considerably greater than expected. In addition, stakeholder exit surveys at the two pilot sites (Selayar and El Nido) found that approximately 97 percent of respondents -- as compared with an appraisal target of 15% (646 percent achievement) agreed or strongly agreed that they had benefited from use of the information on ecosystem services to support decision-making due to their involvement in the project. The overall Efficacy rating is Substantial. Overall Efficacy Rating Substantial 5. Efficiency Even though no economic analysis was undertaken in the Project Paper (and the reasons why this was the case are not explained, but in a subsequent communication with IEG, the author believed that, because of the small size of the project grant, it may not have been required), the ICR presented both an indicative cost-benefit analysis and an implementation efficiency analysis. According to the ICR (para 41, pg. 15), the project's potential economic benefits included the incremental benefits to local communities "from fishing after application of the new tools and information provided by the project in order to prevent the degradation of coastal ecosystems and to improve the efficiency of local businesses that were expected to operate in a more sustainable manner" based on the "critical assumption" that the training and capacity building provided by the project would "result in behavioral change and local willingness to apply the knowledge generated under CCRES." Stating that the potential cost of avoided coral reef reduction in Indonesia was more important than that in the Philippines -- but not presenting the actual quantitative findings in this regard -- and applying a discount rate of 6 percent over a twenty year period, the analysis indicated that the project may have obtained a Net Present Value (NPV) of US$ 20.8 million and an Internal Rate of Return (IRR) of 28 percent under "the optimistic scenario of CCRES-developed tool utilization." The NPV and IRR under the "pessimistic incremental scenario" at a 6 percent discount rate, in turn, were US$ 4.7 million and 13 percent, respectively. For purposes of this analysis, the ICR used actual project financial costs, but could not use observed benefit from improvements in local reef ecology in terms of fisheries, tourism, or coastal development at the actual project sites because this information was not available as the launching of project tools only began at the end of 2017. Instead, the Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) benefits were estimated "based on the relevant literature review that reflect positive impact attributed to sustainable coral reefs use in the Coral Triangle." However, neither the main text of the ICR, nor the Efficiency Analysis described in somewhat greater detail in Annex 4 (pp. 51-53) provided further information in this regard. Nor did they explain the difference between the "optimistic" and "pessimistic" incremental scenarios, which presumably incorporated different levels of the estimated potential benefits. Thus, it is not possible to determine how realistic these indicative estimates may be. On the other hand, the ICR (para 42, pg. 16) also observed, that not all benefits could be quantified with certainty, pointing specifically to "potential benefits from the tools being applied at the regional level or national level in other countries, as well as those potential benefits accruing to local communities from capacity-building that took place during project implementation." Annex 4 (para 4, pg. 52) also affirmed that "the Project positively impacted the socio-economic conditions of beneficiary households," indicating further that "some communities in Selayar Indonesia started to implement community-based tourism through homestays," while "in El Nido, Philippines the CCRES generated some small sustainable business ideas, like using coconut as an alternative to mangrove wood; eco-tourism in the mangroves; cultivating giant bamboo to reduce logging in native forests; and supplying ornamental native flowers to the tourism industry. However, because of the lack of information on actual business development linked to the CCRES implementation, these benefits cannot be estimated." This notwithstanding, this indicative analysis suggests that actual project economic benefits may have been reasonably significant, with the NPV still being positive (US$ 1.5 million) at a 10 percent discount rate even under the "pessimistic incremental scenario." In a subsequent communication with IEG, the ICR author provided the full data set used for the ex-post economic analysis, including the difference between the optimistic and pessimistic scenarios, which, as suggested above, were used for purposes of sensitivity analysis. As to the implementation efficiency analysis, the ICR concluded (para. 43, pg. 16) that "the project performed well" and "met and or exceeded all its objectives as set out in the PDO, and achieved them cost-efficiently, either at or below all estimated Component costs except for Project Management." The project was also implemented within the planned time horizon and closed as expected in December 2018. However, staff turnover both in the Bank's project team (see Bank Performance below) and in the Project Executing Agency (PEA) may have contributed to slower than desired implementation, while delays in initiating some of the uptake activities due to longer than anticipated time for tool finalization may have resulted in "some missed opportunities for CCRES to have early points of entry in programs that could have leveraged and applied tools for impact at a broader (regional and global scale)." The above considerations not withstanding and even accepting that the benefits could have been even greater had uptake occurred earlier during project implementation, the ICR was not able to present evidence based on actual (as opposed to projected) project benefits in light of the critical assumption cited above. Thus, overall Efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) 0 Appraisal 0  Not Applicable 100.00 ICR Estimate  13.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Because the Relevance of Objectives is rated High and the Efficacy of both objectives is rated Substantial, but Efficiency is rated Modest, according to IEG guidelines, the overall outcome of this project is rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome According to the ICR (para. 87, pg. 26), the main risk to the development outcome is the sustainability of model uptake beyond project closure and rated this risk Substantial. However, in reality, there has already been substantial uptake of these tools, and again according to this source a "core focus" of the Bank's supervision during the last two years of project implementation was "to identify a set of platforms to adopt and house the models developed to ensure sustainability ad continue promotion of uptake." The University of Queensland (UQ), the University of the Philippines Marine Science Institute (UPMSI) and PEMSEA (Partnership in Environmental Management for the Seas of East Asia) were expected to play an essential role in this regard, while a number of other organizations have agreed to continue to apply CCRES models in their activities and UQ also continues to promote uptake through its engagement in the policy and academic spheres. Thus, the institutional, financial, technical, economic, social, and other risks to the development outcome of this project would, in fact, appear to be modest. 8. Assessment of Bank Performance a. Quality-at-Entry Considering that this was a small TA project, Bank performance to ensure quality at entry was generally Satisfactory. Project design was appropriate and sought to fill knowledge gaps that had been identified on the basis of previous efforts to support coastal and marine biodiversity protection and fisheries management and from associated Coral Reef Target Research (CRTR) in the Coral Triangle region, which contains some of the world's most abundant and diverse coral reefs. However, no ex-ante economic analysis or incremental cost analysis was undertaken during project appraisal. In addition, according to the ICR (para. 84, pg. 26), "design activities could have more seriously considered Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) comparative advantages of individual project players to determine who was best placed to play a leading role in delivering on the 'science-to-action' demonstration with mechanisms in place to ensure delivery along a predetermined timeline" while "integrating these design aspects upfront could have amplified CCRES impact in the region by providing timely scientific input to COREMAP-CTI, the Philippines Rural Development Project (PRDP), WAVES (Wealth Accounting and Valuation of Ecosystem Services), PEMSEA, and other relevant initiatives in the region to show impact at scale." Quality-at-Entry Rating Satisfactory b. Quality of supervision The project was likewise generally well supervised, although in the words of the ICR (para. 85, pg. 26), it experienced some "inefficiencies" due to the turnover of task team leaders (TTLs), of which there were three -- one for preparation and appraisal and two others over the course of the five year implementation period. On the other hand, the supervision missions were reportedly "promptly conducted," "well-staffed," and "provided targeted support" for most of the project's duration, while also "proactively" identifying issues and working closely with the counterparts to address them." More "hands-on" field-based supervision, moreover, occurred following team leadership changes midway through the project period, which contributed to the acceleration of implementation and avoided the need for any restructuring. On balance, therefore, the quality of supervision is likewise rated Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design According to the Project Paper (paras. 40-41, pg. 47-48), a M&E system would be developed during the first year of implementation and would consist of baseline data including on GEF International Waters Tracking Tool Indicators, in order to track CCRES's progress on an annual basis. The Key Results Framework contained in the Project Paper, including the PDO and Intermediate Results Indicators, would provide the basis for this system, and the PEA would have responsibility for monitoring and evaluating project implementation and outcomes. It should be noted; however, that two of the PDO indicators -- "innovative models developed at two or more sites by the project, demonstrating explicit links between ecosystem health, the value of ecosystem services, and their distribution among stakeholders" and "develop new of improve existing business models directly or indirectly linked to coastal marine zone ecosystem services in at least two sites" -- refer more to project outputs than to outcomes. The PEA was also expected to monitor any unforeseen environmental or social impacts together with the effectiveness of Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) any mitigation measures if required. It was likewise expected to carry out impact assessments at the mid- term and project end dates to determine project achievements in relation to its development objectives. The mid-term review and client's completion report were expected to be undertaken by an externally engaged consultant, and annual progress reports would be submitted by the PEA. b. M&E Implementation M&E implementation was reportedly (ICR, para 74, pg. 24) "robust, consistent, of high quality, and systematically tracked progress towards achievement of the PDO." While baseline information was already available to a significant extent at El Nido (Philippines) prior to project implementation, a greater effort was required to gather such information at Selayar (Indonesia). Where possible, gender disaggregation was undertaken in the feedback surveys carried out at both project sites. c. M&E Utilization M&E information was fully utilized and proved effective for project monitoring, coordination, and reporting. It also helped to incorporate stakeholder feedback into the scoping and design of models and tools under development in order to make them user-responsive and tailored to the local context. M&E Quality Rating Substantial 10. Other Issues a. Safeguards According to the Project Paper (para 39, pg. 47) even though CCRES was essentially a "Research and Innovation/Knowledge Management Project, with a very low probability of any adverse social or environmental impacts, it was classified as Category B for safeguard purposes. This source also indicated that the project, which was expected to generate "substantial net economic and social benefits to stakeholders," had been designed "to reduce the risk of any adverse social impacts through stakeholder mapping and analysis of benefit flows from ecosystem services at t0 [time zero] and then over time to evaluate the consequences of different scenarios -- who wins and who loses." According to the ICR (para 79, pg. 25), the project triggered OP 4.01 (Environmental Assessment), OP 4.04 (Natural Habitats), OP. 4.10 (Indigenous Peoples), and OP. 4.12 (Involuntary Resettlement) and the only noteworthy safeguards issue during implementation was "the requirement for access to appropriate guidance reflecting the World Bank's OPs for end users." In addition, the Bank's safeguards team supported the PEA in preparing simplified safeguards guidelines that were disseminated to end users with all project tools. While the ICR does not explicitly state that all safeguards requirements were complied with it does observe that "aside from one instance of Moderately Satisfactory pending the finalization of these guidelines, CCRES was consistently rated Satisfactory in terms of safeguards performance." In subsequent communications with IE, both the last TTL (August 8, 2019) and the ICR author (August 7, 2019) confirmed that safeguard requirements were fully complied with, Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) b. Fiduciary Compliance Procurement: No problems with procurement were reported and performance in this regard was rated Satisfactory throughout project implementation. According to the ICR (para. 81, pg. 25), the staff of the University of Queensland (UQ) was "well-qualified" to meet the fiduciary standards required for Bank- financed projects. Financial Management: Performance in this regard was also consistently rated Satisfactory during implementation and cumulative disbursements were in line with the expected rate. Subsequent communications with IEG from the ICR team (August 7, 2019) confirmed that project fiduciary aspects were straightforward and handled properly by a specialized and experienced unit at UQ which had very capable staff. Audit reports were submitted on time and were unqualified. c. Unintended impacts (Positive or Negative) The only unintended impact cited in the ICR (para. 59, pg. 20), the models developed under the project were being "leveraged as didactic tools in the university curriculum" at Palawan State University (PSU) even though project design "did not intend to develop model for application in provincial educational institutions." d. Other Gender: The project contributed to local women's empowerment as a result of their participation (as 47 percent of the estimated 3,001 direct project beneficiaries) in project focus groups, workshops, and training activities at the two pilot project sites. According to the ICR (para. 50 pg, 18), anecdotal evidence suggested that the project played a role "in enhancing women's voice and agency in day-to-day decisions around how communities interact with and leverage local ecosystem services." It added, for example, that "one female workshop respondent noted that, for women in coastal locations, the hands-on capacity building provided during model uptake was the first time that they had experienced such training and that, for many, it was their first interaction with any formal education or training effort. This is particularly the case of behavioral change workshops and the systems-based thinking models (Systory and SESAMME). The latter were developed and refined in the field and piloted through a rigorous consultation approach in focus group discussions across project sites." No negative gender-related impacts were reported either during CCRES survey activities, during ICR interview feedback, or through the CCRES Grievance Redress Mechanism (GRM), according to the ICR (para. 52, pg. 18). Institutional Strengthening: According to the ICR (para. 53, pg. 19), government institutions and NGOs at both project sites welcomed its interventions "to enhance the knowledge and capacity of staff to better enable them to undertake marine spatial planning activities and to use results from models as the basis for enacting local legislation and land- and marine-based activities." The awareness raising, capacity building, and technical assistance provided by the project reportedly had a "tangible impact on coastal resource management" particularly at the district government and community levels both in Indonesia and the Philippines. In Indonesia, this impact was manifested in the finalization or revision of village-level Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) regulations (PERDES) on coastal resources utilization and management, including finalization of a PERDES for Bungaiya village that had been stalled since 2009. In the Philippines, workshops and direct technical support were provided to Batangas and Palawan and a monitoring approach was used for Lanuza Bay in order to embed capacity in participating organizations so that their staff could be able to independently replicate the use of project-developed tools in other sites. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Both quality at entry and quality of supervision were generally of Moderately good quality despite some Bank Performance Satisfactory Satisfactory reported "inefficiencies," which the ICR does not describe in any detail Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR presented three lessons, two of which summarized below are of broader applicability: 1. The project illustrates that multi-partner, multi-country, multi-stakeholder projects are complex to design and challenging to implement. This is particularly the case for TA projects with limited financial resources. Associated lessons were that: (i) project design should be realistic in terms of what can be achieved during its planned implementation period and (ii) World Bank implementation support needs to be gauged accordingly. In the present case, having a dedicated and experienced implementing agency helped to overcome the challenges posed by these complexities. 2. Both top-down and bottom-up approaches are necessary even in small TA projects such as this one; while the latter may be welcomed by local stakeholders but may be insufficient to achieve longer-term institutionalization. The corresponding recommendation was that projects such as the present one should continue to engage with local communities but also target uptake measures with provincial and national governments to aid institutionalization and demonstrate the positive impacts of the innovative models developed. The grantee's completion report adds several other operationally relevant lessons based on the project experience (ICR, Annex 5, pg. 54), specifically that it is essential to (i) dedicate more time at the beginning of the project to develop a clear activity plan with concrete deliverables per year; (ii) establish feedback mechanisms (surveys) earlier on in the project to improve Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCRES (P123933) tracking; and (iii) address uptake strategies at the beginning of the project by bringing together experts in translation of research and end users to discuss and develop a roll-out plan. Had these lessons been followed, the project might have been even more efficient and effective. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well-written and of good quality. More specifically, the quality of the evidence provided was generally good as was that of the analysis. It was also internally consistent. However, it could have clarified how the number of project beneficiaries -- reportedly around 3,000, compared with an appraisal estimate of 1,000 -- was determined and why this number was three times higher than originally anticipated. As noted above, this was subsequently clarified in a detailed written response to IEG from the ICR author dated August 7, 2019. The ICR should also be commended for undertaking an economic analysis even though it might also have explained why no such analysis was carried out at appraisal and reported in the Project Paper. The associated annex could have provided greater information regarding the results of this analysis for the pilot project sites in Indonesia and the Philippines although this information was later provided to IEG by the ICR team, as was clarification as to how the number of project beneficiaries was determined and why thus number so greatly exceeded the appraisal estimate. Finally, the ICR could also have provided more information regarding the nature of the "inefficiencies" associated with the turnover in Bank TTLs and Project Executing Agency (PEA) staff, which, in any event, did not result in a need to extend the project closing date. a. Quality of ICR Rating Substantial Page 17 of 17