IEG Report Number: ICRR14641 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 04/14/2015 Country: Indonesia Project ID: P071296 Appraisal Actual Project Name: Urban Sector Project Costs (US$M): 63.4 52.5 Development And Reform Project L/C Number: L4786; L7760 Loan/Credit (US$M): 45.0 36.3 Sector Board: Urban Development Cofinancing (US$M): 5.0 4.5 Cofinanciers: Japan Board Approval Date : 06/07/2005 Closing Date: 06/30/2011 05/31/2013 Sector(s): General industry and trade sector (69%); General transportation sector (22%); Central government administration (4%); Other social services (3%); Sub-national government administration (2%) Theme(s): Urban services and housing for the poor (50% - P); Municipal governance and institution building (50% - P) Prepared by: Reviewed by: ICR Review Group: Coordinator: Roy Gilbert J. W. Van Holst Christopher David IEGPS1 Pellekaan Nelson 2. Project Objectives and Components: a. Objectives: Original: “To assist the Borrower in improving the provision of urban services by Participating ULGs (Urban Local Governments) and Additional Participating ULGs .” (Loan Agreement - LA (Schedule 2, p. 20) and Project Appraisal Document - PAD (p. 8) More details about the project's objective are provided by the PAD (pp. 8-9), and they are consistent with the LA objective as follows: (a) improve municipal governance through enhanced civic participation in key municipal decisions and monitoring/supervision of their implementation, adoption of extensive public information disclosure policies, and reform of procurement and financial management practices. (b) strengthen municipal institutional capacity to formulate long-term urban development strategies and plans, including local economic development and urban poverty reduction strategies. (c) build municipal institutional capacity and professionalizing municipal managers and staff. (d) enhance fiscal capacity by rationalizing expenditures and increasing revenues. (e) finance priority urban investments. (f) implement at the central level, the Urban Institutional Development Program (UIDP) for supporting urban reforms and institutional capacity building. Revised (May 2010): “To assist the Borrower to strengthen local governance and improve the provision of selected urban services by participating ULGs and additional participating ULGs .” (LA 2010 Amendment p. 3) There were two small changes: (i) "municipal governance" , previously highlighted by the specific PAD objective (a) above, was incorporated explicitly into the revised overall objective by referring to "local governance"; and (ii) the qualifier "selected" was inserted to acknowledge that the project no longer intended to improve all urban services, but left the criterion of selection unspecified. Under the original objective, the project made 36.4 percent of the total Loan disbursements. The remaining 63.6 percent were disbursed under the revised objective. For a project with a revised objective such as this, IEG/OPCS procedures call for split ratings for each objective, and using the shares of disbursements noted here to estimate an overall project outcome rating from the split ratings. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval: 06/04/2010 c. Components: Part A: Urban Reform (appraisal cost US$8.1 million; actual cost US$10.3 million ), including: (i) Strengthening ULG and national government institutional capacity through: (a) technical assistance and training for ULGs to carrying out core governance reform programs in the areas of civic participation and transparency, public procurement and financial management; the preparation and implementation of long-term urban development strategies and plans, including local economic development and urban poverty reduction strategies, and carrying out institutional and capacity development programs identified by ULGs; and (b) technical assistance to central government ministries and state agencies for studies to improve national urban strategy formulation, the development and implementation of national urban reform programs, and their capacity to support urban reform in Kabupatens and Kotas; and (ii) Strengthening the institutional capacity of the Ministry of Public Works (MPW) to carry out its administration, supervision and monitoring responsibilities under the Project. Part B: Urban Investment Support Program for ULGs (appraisal cost US$39.3 million; actual cost - US$42.2 million), including sub-projects in participating ULGs financed by Project sub-loans. Sources: Description of components from LA Schedule 2 (p. 20); appraisal cost (without contingencies and taxes) from PAD (p. 37) ; actual cost from ICR (p. 41) At appraisal, these components were to be implemented by 13 ULGs selected during project preparation for their commitment to reform. Shortly after appraisal, however, 8 of the 13 withdrew from the project at their own volition, mostly through shifting priorities. Several years later, 5 other ULGs were signed up to replace the 8, making a final total of 10 ULGs hosting project components by completion (ICR p. 9). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost Actual project costs at closing, at US$52.5 million, were 83% of the appraisal estimate of US$63.4 million including taxes of US$4.7 million and contingencies of US$9.1 million. These contingencies, together with an "unallocated" amount of US$2.0 million, accounted for 17.5 percent of appraisal cost estimates, an unusually large share for an investment project in a well-known sector in a client country very familiar to the Bank. The "unallocated" amount of US$2.0 million reported by the PAD (p. 37) was not the appropriate cost category since, in fact, the amount constituted part of the contingency estimates for the project investments. Financing Unusually, Bank financing of the project was, formally, through two IBRD Loans, each committing US$45 million funding. Through cancellations of amounts in both Loans and a conversion of an amount from one Loan to the other, the total commitment of the two was reduced from US$90 million to US$45 million as follows:  Loan 4786 for US$45 million approved June 2005/closed May 2013. US$9.3 million disbursed (and converted to Ln7760); US$35.7 million cancelled.  Loan 7760 for US$45 million approved July 2009/closed May 2013. US$27.1 million disbursed; US$9.3 million converted (from Ln4786); US$6.9 million cancelled; US$1.8 million undisbursed. The ICR does not explain or evaluate why this unusual financing arrangement was applied to this project. In terms of the amounts and timing of Bank financing it appears to have been no different than if the project had been financed by a single Loan. Later, the project task team informed IEG that the second loan arose after the Bank had agreed in June 2009 with the Borrower to update all Bank loans in the Indonesia portfolio, not just for the present project, to incorporate fixed spread pricing. This was with a view to lowering the costs of Bank loans to the Borrower. Of the total net Bank funding commitment to this project of US$45 million, 81%, or US$36.3 million, was actually disbursed by completion. According to the ICR, US$6.9 million was cancelled from the Loan because of the fewer ULGs participating (ICR p.10). Unusually in this case an amount of US$1.8 million that remained undisbursed was not cancelled. Cofinancing through a PHRD grant committed by Japan in an amount of US$5.0 million, almost 10 percent of the total project cost, resulted in spending of US$4.5 million. This trust fund was administered by the Bank. While the ICR reports (p. 8) that it was to be used to finance the Project's Part (A) Urban Reform component, the ICR does not evaluate what it was spent on, how well it was used, nor why an unspent amount of US$0.5 million remained at project closing. According to comments received later by IEG from the project task team, the information on the use of the fund was not available in time for the ICR. The team also later informed IEG that this grant money was used to finance 21 workshops to deliver training for capacity building at the national and local levels. The national project executing agency, Cipta Karya did not use all the grant funding as it has its own budget resources from Government. Nevertheless, according to the task team, all the planned TA activities were delivered. Borrower Contribution By completion, the Government of Indonesia paid into the project US$1.9 million in counterpart funding, more than twice its commitment of US$0.9 million made at appraisal. The 10 participating ULGs, on the other hand, paid in less than 13 ULGs originally signed up at appraisal had promised. By completion, the 10 had paid in US$10.0 million, somewhat short of the US$12.5 million appraisal commitment by the 13. On a per ULG basis, the amounts committed and actually paid in were similar, however. Dates The actual closing of the project in May 2013 was two years later than planned at appraisal. The loan closing had been formally extended two times owing to implementation delays from start up. Because of these delays only 59% of the original Loan commitment had been disbursed by the project's original closing date of June 30, 2011, notwithstanding the project restructuring that took place on March 18, 2010 when only 36.4 percent of the Loan had been disbursed. 3. Relevance of Objectives & Design: a. Relevance of Objectives: rated - Modest The project's objective was directly relevant to the Local Services Platform of the 2003 Country Assistance Strategy (CAS 2003) in force at the time of the appraisal of this project. The Local Services Platform, one the CAS 2003's four, was an answer to the challenges for service provision at the district level following Indonesia's 2001 decentralization, precisely the focus of this project's (LA) objective (CAS 2003 pp. 30-31). This objective remains relevant currently to the 2012 Country Partnership Strategy (CPS 2012) for the FY13-FY15 period, that highlights improved local government service delivery and strengthening governance through better fiduciary, social and environmental management and technical performance as elements of its Pro-Growth Strategy, an effort directly supported by the project (CPS 2012 pp. 18-19). As far as priorities and strategies on the Government side are concerned, the ICR does not assess the relevance of the project objectives, although it is clearly understood that the project was relevant to Indonesia's 2001 decentralization policy that remains in place today. The ICR does not refer to any later Government policies or documents, notably at completion, to which the project objectives may or may not still be relevant. Later, however, the project task team informed IEG that the project objective remains relevant to Indonesia's current sector priorities as stated in Bappenas' 2012 National Urban Strategy and Policy Development (NUSPD) that highlights the increasing role and responsibilities of ULGs in urban service delivery. Regarding the formulation of the objective itself, the ICR does report that it was not as clear is it could have been in defining the intended outcomes of the project (ICR p. 12). This lack of clarity was exacerbated by the inconsistent emphases, noted earlier by this review, between the LA version, built around improving the provision of urban services and the PAD version, built around improving urban services themselves. The use of both in the PAD's Results Framework also did not serve to clarify what the project was trying to achieve. With an objective of improving the provision of urban services, the project would have supply-side focus upon the suppliers of these services, as it did, in practice, for the most part, through the ULGs. With an objective of improving urban services, the project would have to give more attention to the much broader demand-side results, from the perspective of the users of these services. The May 2010 revision of the project objective that brought governance improvements to the forefront, did not resolve this ambiguity. The PAD version remained 'alive' and unchanged until project closing. Whichever version one considered, however, it is clear that the huge scope of the objectives in seeking general improvements either to urban service delivery, or to the conditions of urban services themselves in a large country like Indonesia, was not consistent with the small scale of an operation focussed upon a few eligible ULGs. A more modest objective, such as, testing pilot revenue generating approaches to improvements to local urban service delivery, would have been more fully achievable. IEG later learned from the project task team that the Government insisted on retaining what it called the "open menu" approach of the project objective, which misleadingly indicated that all urban services would be eligible for project support. The Bank's response of introducing the term "selected" to qualify the eligible services in the revised objective did not resolve the ambiguity of the project's declared intent on the one hand and its real capability to deliver on the other. b. Relevance of Design: rated - Modest The ICR does not directly assess the relevance of design, understood to be the consistency of the project components with the project objective. Nevertheless, IEG considers that the first component, Urban Reform, directed at strengthening the governance and management of ULGs with growing service responsibilities under decentralization, could contribute directly to improving the provision of urban services as the project intended. Project technical assistance and training covering, among other things, procurement and financial management with which these ULGs were unfamiliar, could strengthen ULGs as urban service providers as the project objective intended. The second component, the Urban Investment Support Program could, through on-the-job learning by managing the project's own investments, lead to service improvements. But the improvement would depend upon the project's requirements for ULG service delivery, the relationship to the Urban Reform component, the types of services eligible, and the roles and responsibilities of different levels of government in their delivery. In hindsight, these could have been made clearer in the project design. On the project's Results Framework , the ICR appropriately made several critical comments. Baselines and targets were not quantified (ICR p. 4). It did not show how one project component's results affected the other's (ICR p. 5). It was not updated after the 2010 revision of the project objective to upgrade intermediate outcome indicators of governance indicators to become full outcome indicators (ICR p. 6). In spite of these criticisms, the ICR concludes that the Results Framework indicators were appropriately linked to the project objectives (ICR p. 19). Beyond the ICR criticisms of the Results Framework, IEG notes some additional weaknesses. An important one already mentioned is that the Results Framework at appraisal incorporated both (inconsistent) versions of the project objective, one directed at improving urban services (LA), and the other at improving the provision of urban services (PAD p. 22). The ICR itself falls victim to this confusion when it misquotes the Results Framework statement of the project's outcome indicator. The ICR omits the crucial word "delivery" found in the PAD's Results Framework when reporting that indicator (ICR p. 4). Another major weakness of the Results Framework was in having only one outcome indicator ("at least 10 ULGs improve priority urban service delivery selected for project support" (PAD p. 22) that was not really an outcome indicator. The ICR itself correctly recognizes this just an input indicator (ICR p. 4). The Results Framework does not explain the links between implementing the project and achieving the improved urban service delivery that it sought, the logic of which being something that a results framework is supposed to illustrate. The lack of explanation is partly due to the very summary form of the appraisal Results Framework for this project. It covers barely one and a half pages of the PAD, is almost devoid of quantitative references, and lacks clear and precise descriptions of the expected results. 4. Achievement of Objectives (Efficacy): IEG's present review considers the efficacy in achieving both the original and revised Loan Agreement objectives, giving each one due weight according to the respective shares of disbursements they covered. The review also assesses the achievement of the relevant specific objectives defined in the PAD which were consistent with those of the Loan Agreement. Objective of the Original Loan Agreement (covering 36.4% of disbursements) “To assist the Borrower in improving the provision of urban services by Participating ULGs and Additional Participating ULGs” and achieve the following specific objectives listed in the PAD that are consistent with this objective: (a) improve municipal governance through enhanced civic participation in key municipal decisions and monitoring/supervision of their implementation, adoption of extensive public information disclosure policies, and reform of procurement and financial management practices; (b) strengthen municipal institutional capacity to formulate long-term urban development strategies and plans, including local economic development and urban poverty reduction strategies; (c) build municipal institutional capacity and professionalizing municipal managers and staff; (d) enhance fiscal capacity by rationalizing expenditures and increasing revenues; (e) finance priority urban investments; and (f) implement at the central level, the Urban Institutional Development Program (UIDP) for supporting urban reforms and institutional capacity building. rated - Modest Positive results partially achieved through:  better ULG governance, involving public participation (albeit on a smaller scale than intended) that could improve service provision  modest ULG efforts at planning that could provide a basis for service provision  improvements in ULG financing which may be related to service provision. Shortcomings include:  actual scope of potential beneficiaries was much smaller than appraisal intent  all sub-projects actually implemented but one, are urban markets, a far narrower scope than the broad urban services intended across various sectors. Revised objective (covering 63.6% of disbursements) “To assist the Borrower to strengthen local governance and improve the provision of selected urban services by participating ULGs and additional participating ULGs . rated - Modest Positive results partially achieved through:  better ULG governance, involving public participation that could improve service provision  modest ULG efforts at planning that could provide a basis for service provision  improvements in ULG financing which may be related to service provision. Shortcomings include:  actual scope of potential beneficiaries was much smaller than appraisal intent  all sub-projects actually implemented but one, were urban markets, a far narrower scope than the broad across-sector urban services intended across various sectors Specific Objective s in PAD consistent with those of the Legal Agreements (a): to improve municipal governance through enhanced civic participation in key municipal decisions and monitoring/supervision of their implementation , adoption of extensive public information disclosure policies , and reform of procurement and financial management practices . rated - Modest Outputs:  90 percent of the 10 final participating ULGs met the project's five requirements that they post updates on line or in print media on: (i) local development activities and programs; (ii) the mid-term development plan; (iii) a summary of the local budget; (iv) audited annual financial report; (v) procurement plan for the current fiscal year and contracts awarded in the past six months. Although the project met targets expressed as the number of ULGs "affected", actual civic participation, expressed by the population involved, was much less than the project intended--only 40 percent of number of beneficiaries promised by the project. This is for two reasons. Fewer ULGs actually participated in the project than intended, 10 instead of 13. The average population of the 10 was much smaller than of the 13. In hindsight, the absolute number of people benefitting might have been a more meaningful indicator.  All 10 final participating ULGs report having public "feedback mechanisms" in place with at least one type of media for public complaints managed by a specific complaint handling unit, as well as standard operating procedures for handling public complaints. While this result exceeded the target of 90 percent of ULGs with these mechanisms, useful output information about them was not provided. This would include how often, if at all, the mechanisms were used, by whom, and for what purposes--how often for urban services, for instance--and the follow up actions, if any, taken by the ULGs. Even with that information, these actual feedback mechanisms would have benefitted a far smaller public than intended at appraisal, given the fewer and smaller ULGs that actually participated, as just mentioned. Outcomes:  An indirectly related outcome comes from the findings of beneficiary surveys of 60-200 traders and consumers per location showing that in 60 percent of ULGs, at least 50 percent the respondents were satisfied with the (market) facility provided by the project, a very low satisfaction bar, perhaps, in the eyes of some reviewers. The project's target had been 70 percent of ULGs reporting with this 50 percent level of satisfaction. As outcome evidence, questions may be raised about its quality. How were the respondents chosen and what questions were put to them? The ICR does not provide details. Also, how was it possible to report the levels of satisfaction of presumably potential traders and potential consumers with three market facilities that were not in use? How was it possible to relate these respondents to these unused facilities, and what can their "satisfaction" mean regarding a facility that is not used by them, or anybody else? Such findings, when still no services were being provided, do not offer convincing evidence of the project contributing to achieving its overall objective of improving the provision of urban services.  80 percent of the 10 participating ULGs adopted a participatory planning process for allocating block grants or for budgeting, against a target of 50 percent. As before, evidence of how the process was being put into practice would have made this a stronger finding.  All ten participating ULGs issued local regulations as part of sound management practice, against of target of 80 percent.  90 percent of the 10 final participating ULGs awarded bids within the required period and all their tender committees were formally validated. ULGs received project technical assistance and training to help them achieve this. b): To strengthen municipal institutional capacity to formulate long -term urban development strategies and plans , including local economic development and urban poverty reduction strategies . rated - Modest Outputs:  All 10 participating ULGs developed Local Economic Development (LED) plans, against a target of 60 percent.  The ICR does not report on the results of formulating City Development Strategies (CDS) that were similarly to be developed. CDSs are flagship, stand alone, urban planning products supported by the Cities Alliance global program partnership. This is how they were introduced in the PAD (PAD p. 5). Instead of clearly reporting how many, if any, ULGs had prepared CDSs, the ICR treats CDSs in the Results Framework and elsewhere (ICR p. 27) as if they were inputs to studies of local economic development (LED)  Similarly the ICR does not report on ULG formulation of Poverty Reduction Strategies that they were expected to prepare.. Outcomes:  (Evidence of achieving the objective of strengthening municipal institutional capacity would require baseline information, not provided in the ICR, about the existing institutional capacity in the ULGs prior to the project to formulate these strategies and plans.). (c): To build municipal institutional capacity and professionalizing municipal managers and staff . rated - Modest Outputs:  Support to 52 training events and workshops between 2006 and 2013.. Outcomes:  (No specific evaluation of the results is reported by the ICR). (d): To enhance fiscal capacity by rationalizing expenditures and increasing revenues . rated - Modest Outputs:  Technical assistance and training provided to ULGs. Outcomes:  The ICR reports that 40 percent of the 10 final participating ULGs increased their, presumably annual, revenue by 10%, against a target of 70 percent of ULGs.  70 percent of the 10 final participating ULGs increased their expenditures on operations and maintenance (O&M) by at least 10 percent, meeting the target of 70 percent (ICR p.28). Although this is not completely clear in the ICR, it would appear that this result applies only to facilities in the ULG directly supported by the project. In that case, the impact of this result might not be significant at the level of the ULG as a whole, or attributable at this level to the project. The project objective seeks improvements for the group of 10 at the level of the ULG as a whole. (e): Finance priority urban investments . efficacy rated - Modest Outputs:  The ICR reports the delivery of 15 sub-projects in 10 ULGs at an average cost per component of US$2.8 million. Of the 15, 14 were for markets and one was for a bus terminal. At the time it was issued in November 27, 2013, the ICR informed that, of the 15, 5 were "fully used", 5 "partially used" and 5 "not yet used" (ICR p. 46). Through contacts with the project task team, who had inquired of the ULGs concerned, IEG received the following updates on the use of these facilities. They show significant increases from the ICR figures in the share of all facilities in full use. They also show that the number of facilities still unused fell from the ICR's 5 to, most recently, 3. Updates of Reported Use of Project Facilities Source: Date: fully used partially used not yet used total: ICR Nov 27. 2013 5 5 5 15 TTL email 1. July 2, 2014 5 7 3 15 TTL email 2. Jan 8, 2015 11 1 3 15 Outcomes:  The ICR does not explain why or how project investments in markets correspond to the "priority" investments in urban services that the project intended to support. The project design, sustained by the references to urban services in general by all formulations of objectives--the original LA and PAD versions as well as the revised LA version of 2010--had in mind priority investments in roads, bridges, water supply, sewerage, solid waste management and flood control, as well as the urban markets in which almost all project investments were made. By authorizing only direct revenue generating investments as being eligible for project financing, the Government precluded project investments in urban services across these various sectors under the prevailing conditions of urban service provision in Indonesia (ICR p. 9). (f): To implement at the central level , the Urban Institutional Development Program (UIDP) for supporting urban reforms and institutional capacity building . rated - Substantial Outputs:  Establishment of a Center of Excellence for Urban Management (called PIP28 in Indonesia)  Local Economic Development (LED) studies developed by each ULG participating in this project.  Support for the drafting of Bappenas' National Urban Strategy and Policy Development (NUSPD) and five ULGs' Local Urban Strategy and Policy Development (LUSPD).  Deployment of TA consultants to participating ULGs to help them implement their reform agendas.  Consultants to assist national executing agency, Cipta Karya, to oversee the reform agenda at the local level. Outcomes:  Following the Sulawesi experience in this project, 32 other provinces established Centers of Excellence specifically focused upon disseminating best practice examples related to key aspects of urban management (e.g. in Bali for traditional buildings, and Yogyakarta for disaster risk management). Actual scope of project results versus their intended scope The scope of the project's actual results at completion was considerably reduced when compared with scope of the results sought at appraisal. At appraisal, the project aimed to assist 13 pre-selected ULGs administering a total population of 6 million. By completion, the project assisted 10 smaller ULGs covering just 2.4 million people, according to the 2010 census. The sharp fall in the actual beneficiary population resulted from the withdrawal of eight of the 13 larger ULGs originally selected, and their replacement by five smaller ones. Almost all project investments completed by the final closing were in markets, against an initial broad concept of urban services conceived at appraisal that included: improved roads, bridges, bus terminals, water supply and sewerage infrastructure, solid waste management and flood control (PAD p. 56). Particularly when reporting actual economic benefits and revenues generated by project investments, the ICR cites these sectors as key sources of them (pp. 52-53) 5. Efficiency: rated - Negligible The assessment of this project's efficiency was weak. There were few concrete data on costs and benefits both at appraisal and completion. The PAD did not report economic rates of return (ERRs) of the proposed physical investments that could have been evaluated in this way. Instead the PAD (generically) affirmed that the project's "urban investments are expected to have major economic benefits, such as: (a) reduction of travel time and costs through improved roads, bridges and terminals; (b) reduction in health hazards through better water supply and sewerage infrastructure; and (c) improvement in environmental conditions through solid waste management and flood control, to cite a few examples." (PAD p. 56). While true in general, these benefits could not come from a project nearly all of whose intended (and actual) investments were only in markets--already known to be the case at the time of appraisal. The PAD reports separate FRRs of 18 sub-projects (14 markets, 3 bus terminals and one swimming pool) totalling U$47.3 million, that combined give a weighted average FRR (estimated by IEG) of 10.4 percent (pp. 28-31). In hindsight, the ICR considers that several of these FRRs were optimistic (ICR p. 35). All the PAD estimates were, nevertheless, above 8.19 percent, the then prevailing rate of on-lending of project funding to the ULGs (p. 56). The PAD and ICR analyses do not specify what costs and benefits led to these estimated FRRs, or what parameters were used to measure them. ERRs were not estimated at appraisal, even though the PAD required a minimum 10 percent ERR for larger subprojects (>US$550,000) to be eligible for project funding (p. 56). For the economic and financial analysis of the project, the ICR reports that recalculating FRRs at completion was not done because sufficient field data were not available (ICR p. 30). The ICR only provided an assessment of one small, partially used sub-project, the market in Sawahlunto, rebuilt at a cost of US$1.8 million, 3 percent of final total project costs. It had no comparable ex ante evaluation because it was not among the ULGs or sub-projects identified at appraisal. The ICR makes no claim that it is representative of the wider project. These factors preclude the use of this analysis alone to assess the efficiency of the project as a whole. That said, the results reported by the ICR for this project included: NPV (net present value) of 8,060 million rupiah (with a discount rate of 11.5 percent) and an FRR of 18 percent. Costs include investment costs plus annual O&M cost throughout the 20 year life of the sub-project and benefits are derived from revenues from various sources (ICR p. 54). The ICR mentions "average cost effectiveness of investment subprojects" as one aspect of the project's overall efficiency assessment, but does not provide data or results for the project as a whole, or a representative sample of it (ICR p. 30). Project implementation and use of its investments by beneficiaries were not efficient. It took nearly two years longer to implement than planned. Disbursements were delayed throughout implementation. By the original mid-term of April 2008 only 16.7 percent of the Loan had been disbursed; by the original closing date of June 2011 only 58.4 percent was disbursed; and by the extended closing date only 80.7 percent had been. The ICR's Disbursement Profile (Data Sheet) indicates a much lower disbursement still. IEG's assessment here is based upon data from the Bank's Client Connection database Regarding the use of the 15 completed facilities, the ICR reported as of November 2013 that only five, accounting for 23.8 percent of total costs, were "fully used" (ICR p. 46). It is important to note here that, only a "fully used" sub-project can generate the scale of revenues needed to demonstrate the possible financial and economic feasibility of the investment. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 10.4% 73% ICR estimate Yes 18% 3% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: With relevance and efficacy rated Modest, the latter because of the sharply narrowed sectoral scope and beneficiary coverage of the actual project results, and efficiency rated Negligible for lack of demonstrated results for most of the project, only one quarter of which was in full use at completion, there were major shortcomings in the extent to which this is a project that achieved its objectives either before or after the first restructuring. Derivation of overall outcome from split ratings Since the overall outcome rating of this project was the same before and after the revision of the objective, the consideration of partial ratings, summarized below, did not affect the final rating. Actions prior to the revision of the project's objective when 36.4 percent of disbursements led to an overall outcome rating of Unsatisfactory (value: 2). After the restructuring, when 63.5 percent of disbursements were made, the overall outcome rating is still rated Unsatisfactory (value 2). The final rating (reported below) derives from mean of the pre- and post-restructuring ratings, weighted by the share of total disbursements behind each one, as per the following: (2 x 0.364) + (2 x 0.635) = 0.73 + 1.27 = 2.00, the value of Unsatisfactory. a. Outcome Rating: Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Factors working against the likelihood of this project improving the provision of urban services and strengthening municipal governance into the future include: 1. the absence of data and indicators demonstrating a solid financial performance of most of the project investments at completion. 2. according to the ICR only five of the project's 15 investments were in full use at completion, providing little indication of the likely demand for the remaining market facilities. Information provided to IEG after project closure points to a fuller use of the facilities. 3. likely continuation of central Government decentralization policy focused primarily upon revenue generating services may not directly affect the broader non-revenue generating urban services covered by the project's objective, such as roads, water and sanitation, and solid waste management. 4. uncertainty about most ULGs' ability to provide better urban services into the future, for lack of financial information about their revenues and their ability to finance their own priority investments especially. The ICR reports that only 4 ULGs increased their revenues. 5. volatile ULG commitments to the objectives espoused by the project. Of the original 13 ULGs signed up to the project at appraisal, 8 withdrew of their own accord. In their place the project was able to sign up only 5 smaller ULGs. 6. As the ICR notes, some participating ULGs have yet to set up the necessary institutional arrangements to ensure the sustainable operation and management of their project investments (ICR p 39). Factors that may help mitigate risk to development outcome include: 1. 7 of the 10 participating ULGs have increased their O&M budget by at least 10 percent. This may help sustain other existing urban services. On the other hand, the ICR also reports that the ULG may not provide sufficient budget for O&M on a sustained long-term basis, an "endemic problem in Indonesia"). 2. The 10 ULGs that finally participated in the project were all committed to continuing with the operation's reform agenda (ICR p. 65). 3. The ICR reports that, even for yet unused facilities, there are no technical impediments to the utilization of the markets. This overlooks however, that full utilization depends first and foremost on whether there is existing and continuing use of facilities at the offered price, information that was not available in the ICR. More recent information provided by the task team to IEG since the ICR points to an improved performance of the facilities. 4. Nearly all participant ULGs became more transparent through publicly posting plans and budget information and instituting public complaints mechanisms. If they continue to be used effectively, measures such as these can help sustain a stronger popular demand, down to the grass roots level, for the improved provision of urban services by ULGs. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: Several major shortcomings during preparation and appraisal undermined the quality at entry of this project that had been three years in preparation at a cost to the Bank of nearly one million US dollars:  Inconsistent reporting of the project objective in project documents, with the LA's supply-side attention to "improving the provision of urban services" and the PAD's "improving urban services" more generally. The resulting dichotomy, highlighted earlier in the present review, created uncertainty about this project's exact purpose, that the revision of the project objective in 2010 did not clear up.  Regardless of these different formulations, the project objective contained another major shortcoming. Both its LA and PAD versions addressed ULG urban services as a whole, that included roads, drainage, water supply and sanitation, and solid waste management among others. The Project's first eligibility criterion for sub-projects was simply that they should be “infrastructure projects” (PAD p. 28). In practice, however, all but one of the total of 15 sub-projects actually implemented were urban markets, chosen as the only ones that met the Government’s revenue generating requirements. In spite of them, the project objective, in all its variants, continued to address urban services as a whole. Partly recognizing this inconsistency, the revised objective of 2010 that referred to "selected" urban services, albeit without specifying explicitly the criteria of selection. In hindsight, the project purpose would have been clearer and quality at entry strengthened had the objective specified "revenue generating urban services" explicitly.  The principal conclusion of the July 2005 Quality at Entry Assessment by the Bank’s Quality Assessment Group (QAG) was that the project’s objective, with its “lack of clarity" no longer captured the intended outcomes of the project after late revisions to the project design (ICR p. 4)  The agreements during project preparation for most of the original 13 ULGs to participate in the operation seem not to have been well founded. With the withdrawal of 8 of them shortly after appraisal, noted earlier in this review, the project lost more than half of its sub-national clients, including some of the largest ones. The ICR's Additional Annex 11 summarizes the “underlying reasons”: six ULGs withdrew because of changed priorities of their local administrations, newly elected after the project appraisal; one decided to fund its sub-project with its own resources, while another disagreed with MOF over the on-lending conditions (ICR pp. 77-78). The PAD, on the other hand, reported that the original 13 ULGs signed up because of their recognized commitments to municipal reform and for meeting the operation's creditworthiness requirements (pp. 9 and 14). The original project design clearly expected them to remain on board, since it did not provide a simple process for signing up replacements. To do so required protracted preparation, negotiation and agreement to reach a new sub-loan agreement with each new postulant ULG. According to the ICR this was a very time consuming process for which insufficient resources had been allocated (ICR p. 35). Clearly the risk of ULGs dropping out had not been factored into the design. The ICR itself notes that “several risks became evident only during implementation and had to be addressed in an ad hoc manner which put a serious strain on project management” (ICR p. 36). The result was a significant delay to project start up.  A design principle was that the project be open to all interested ULGs in Indonesia that meet project eligibility criteria (PAD p. 9). During preparation in 2004, Cipta Karya had invited all ULGs in Indonesia to consider participating. Altogether, some 160 ULGs expressed interest. From them Cipta Karya drew up a short-list of the 40 most eligible--being reform minded, credit worthy and having no financial arrears with central government. Cipta Karya technical teams visited all 40 ULGs. The 13 most eligible, with the best sub-project proposals were selected. The remaining 27 ULGs constituted a reserve list. The number of participating ULGs, that fell to just 10, was very small compared with Indonesia's total of more than 500 ULGs (405 Kabupaten, or regencies; and 97 Kota, or cities). To have the promised impact upon improving urban services, or provision of them, the project design would have had to be configured on a much larger scale.  The Project design appears to have not taken advantage of the Bank's extensive experience of urban development assistance in Indonesia, that included prior experience with subloan agreements at the local level (ICR p. 16) and even Bank familiarity with the logistical challenges posed by Indonesia's geography. Quality-at-Entry Rating: Unsatisfactory b. Quality of supervision: Because of the flaws in design just mentioned, this was a very difficult operation to supervise. Altogether 12 supervision missions were fielded regularly during project implementation, twice a year prior to the 2010 restructuring and once a year thereafter. Implementation performance ratings were correctly rated Moderately Unsatisfactory between FY2009 and FY2010. when it became a problem project (ICR p.17). Despite the poor implementation, the same supervision missions uniformly rated the achievement of the development objective as Moderately Satisfactory. The differences between these ratings could tell us one of two things. Either the ratings were simply not consistent, or they accurately tell the story of a project that was not being implemented well, but modestly achieving its objectives. For the second half of the project, nearly all ratings were optimistically Moderately Satisfactory, probably stimulated by the idea of the adoption of a revised objective, that in reality, was not so different. Without specifying exactly how much, the ICR notes that considerable Bank supervision resources were devoted to the processes of admitting 5 additional ULGs into the project (ICR p. 36). The implication is that supervision missions were distracted from more pressing supervision work, such as further refinement and revision of the project objective and performance indicators. The ICR reports that supervision missions monitored sub-project implementation and ULG adherence to Bank fiduciary requirements and safeguard policies. This was costly because participating ULGs were scattered across the country (ICR p. 37), although 6 of the final 10 were located Central and South Sulawesi. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: “Government” for the purpose of this review is considered here to be the national level ministries participating in the project’s Inter-Ministerial Steering Committee (IMSC) chaired by Bappenas, Indonesia’s National Planning Board. While not commenting upon specific IMSC members, the ICR notes that Government ownership was strong throughout and stakeholder consultation prior to appraisal were adequate. As noted earlier in this review, Government counterpart funding of US$1.9 million actually paid in was more than twice the amount promised at appraisal. On the other hand, the Government's requirement, made clear shortly after appraisal, that only revenue generation projects be eligible for project financing fundamentally altered the nature of the project interventions. They consisted almost exclusively of building and refurbishing municipal markets. Broader urban service investments originally foreseen were no longer eligible. The ICR reports that the IMSC provided useful oversight and guidance to the project. Without specifying the ideal number, however, the ICR notes that IMSC support would have been more efficient had the Committee been smaller (ICR p. 38). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: National Level Implementation Agencies For the ICR and the present review, the national level Implementing Agency (IA) of the project was Cipta Karya, the Directorate General of Human Settlements in the Ministry of Public Works. Cipta Karya has had extensive experience of working with Bank financed urban projects since the 1980s. According to the ICR, Cipta Karya performed well in helping to prepare the project, but was not ready when implementation began, citing insufficient staffing and delayed consultant appointments to the Central Project Management Unit (CPMU) as evidence of this (ICR p. 37). The ICR also noted that Cipta Karya had “neither the expertise, nor the full mandate, nor the full range of required resources to guide and oversee comprehensive reforms in local governance (ICR p. 13). Over time, however, CPMU performance improved to the point that it became active in matters that were strictly beyond its project implementation mandate. These included its work with Parliament to help ensure the timely release of project funds. Within its conventional mandate, the CPMU worked closely with ULGs on procurement matters in particular (ICR p. 37). Thanks to the additional counterpart funds paid in by the Government, Cipta Karya had twice the funding originally budgeted for its work on the project. ULG Level Implementation Agencies This review assesses the performance of the 10 ULGs that actually participated in the project by completion, 5 of which were among the original 13 selected at appraisal and 5 were added later. Lack of information precludes an assessment of the 8 ULGs that withdrew from the project after originally signing up. The ICR informs that the 10 participating ULGs suffered considerable weaknesses and constraints, and tells us that their performance was “mixed”, without explaining which ULGs did better and which ones did worse, or how and why (ICR p. 38). More generally, ULGs “were not ready for implementation by loan effectiveness”, being unaware of project "requirements" partly through the limited capacity and experience of their available staff (ICR p. 38). ULGs had to devote substantial time to studying Indonesia's new regulations and gauging the authority and discretion they afforded. Also anti-corruption efforts from 1999 onward sometimes construed minor administrative errors by ULGs as malfeasance, instilling a feeling of insecurity and fear among civil servants and discouraging them from being pro-active and taking bold action when needed (ICR p. 17). With the help of the CPMU, however, some ULG learning did take place about new participatory processes and fiduciary requirements. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The details of M&E planned for this project at appraisal were very sketchy. The PAD Annex 3 on the Project's Results Framework and Monitoring notes that project management units (PMUs) at the ULG level would be responsible for data collection, and for packaging it into routine, mostly annual progress reports. The ICR reports that the M&E design assumed that PMUs at the provincial level with prior experience of working with the Bank already had the capability to do this (ICR p. 19). However, the M&E design at appraisal did not indicate the type of data to collect, its periodicity, scope, or how it should be analyzed to demonstrate progress toward achieving the project objective. One example of an imprecise indicator proposed was that "participating ULGs have sound financial and procurement management practices". There is no reference to what should be measured to monitor this, nor any mention of baseline or targets values. The only indicator pertaining to the Project's Urban Investment Support Program component (accounting for 83% of total project costs) was simply that "ULGs prepare and implement subprojects worth US$52.1 million during the project period", in other words, simply implement the project. The June 2010 restructuring of the project included a revision of the M&E system to address the problem of the lack of monitorable standards for the original indicators (ICR p. 20). The PAD's original single outcome indicator was replaced by a set of four measurable outcome indicators with set targets. The PAD's original six intermediate outcome indicators were replaced by eight measurable ones. These represented an important improvement, although the Results Framework still failed to make clear the relationship of meeting the targets set for the indicators to the achievement of the project's objective. The second (revised) outcome indicator of increased ULG spending on operations and maintenance of infrastructure already provided, while a positive result in itself, had nothing to do with improving the provision of urban services by ULGs yet to be provided, something that the project, at appraisal, promised to achieve. Two other (revised) outcome indicators of the satisfaction of the vendors and users with the services provided, measured through surveys of satisfaction, were closer to the project objectives. They would have more convincingly demonstrated an improvement, however, if they had been able to report the levels of satisfaction prior to the project, namely the baseline, with the levels achieved afterwards. In the case of this project, there were no baseline data for either of these indicators. b. M&E Implementation: The ULGs assumed capable of taking on M&E responsibilities did not meet expectations at the outset. The ICR reports that "the frequency and quality of reporting from the local level was very poor on the grounds of insufficient capacities and the absence of supporting consultants" (ICR p. 20). With the deployment of "bridging consultants" the ICR reports that the situation improved. ULGs were required to report crucial issues immediately, although the ICR does not tell us if they did so. ULGs were also required to provide quarterly and semi annual M&E reports. Again, the ICR did not provide information about the actual performance achieved in the delivery and quality of these reports. The project's Central PMU launched a dedicated interactive website through which participating ULGs could upload relevant information/documents, but we are not told by the ICR how much, if at all, it was used for M&E. c. M&E Utilization: Project M&E data was a useful input to CPMU reporting to the Ministry of Finance, the Government department responsible for foreign funded projects, and also to the Bank. The ICR stated that the CPMU used M&E information to provide written guidance to ULGs who, presumably, were encountering difficulties in meeting targets. The problems of implementing M&E as originally designed highlighted the weaknesses of the original system, recommending its re-design to move it closer to a system able to measure the actual outcomes achieved. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: While project documentation reports this to be an environmental category A project requiring full environmental assessment and screening (ICR Data Sheet and PAD p. 1) the PAD notes that all subprojects prepared by ULGs were environmental category B or C (PAD p. 15), a more realistic assessment given the modest environmental impact of the urban markets financed by the project. The reasons for the different environmental categories, namely 'A' at the national level and 'B' or 'C' at the ULG level were not explained in detail. Even so, the ICR reports that ULGs took their environmental obligations under the project seriously, requiring contractors to adopt Bank safeguard standards. On the other hand, the ICR also reported that ULG staff performance was affected by their lack of familiarity with safeguards (ICR p. 39). Nevertheless, no cases of non-compliance were reported. Regarding involuntary resettlement, the ICR noted the temporary relocation of traders using old markets being redeveloped through the project. Although there were delays, most traders were re-accommodated in the new project facilities. The ICR does not report on resettlement plans or compensation paid. The full weight of the Bank’s safeguard policy, did not apply to the temporary adjustments needed when these did not involve the residencies of those affected. The ICR concludes that ULG project staff performance was adversely affected by a lack of experience in dealing with Bank rules and requirements for procurement and safeguards, deficiencies that had been underrated at appraisal with therefore insufficient attention paid to them by the project design (ICR p. 39). b. Fiduciary Compliance: The ICR provides few details but reports no cases of non-compliance. The project’s financial reports were submitted in a timely fashion, as were audit reports, none of which included qualified findings. Managing procurement in accordance with Bank standards proved challenging for ULGs. The ICR notes that ULG staff deficiencies in knowledge about Bank procurement requirements had been underrated at appraisal, and were thus given insufficient attention by the project design (ICR p. 39). Nevertheless, procurement plans were updated regularly and subsequent contracts awarded in a timely fashion. c. Unintended Impacts (positive or negative): d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Unsatisfactory At completion the actual number of Satisfactory potential beneficiaries was less than half the number intended at appraisal, because of the participation of fewer and smaller ULGs than planned. Narrow scope of the results achieved meant that only municipal markets out of the potentially broad array of urban services in general offered by the objective. Finally, there was no evidence for the project as a whole that outcomes were achieved efficiently. Risk to Development Moderate Significant Lack of data showing, at present, a Outcome: solid financial performance for most of the project's investments creates uncertainty about the future demand and supply of the project facilities. Uncertainty about demand comes from the fewer and smaller ULGs participating than intended, the withdrawal of most ULGs who had signed up at appraisal, and from the delayed deployment of the project facilities, only 5 (out of 15) were in full use at project completion (although later information points to 6 more becoming fully operational after the project closed). Bank Performance: Moderately Moderately Several major weaknesses in project Satisfactory Unsatisfactory design and quality at entry, including: (a) inconsistent reporting of the project objective across project documentation (LA, PAD and Results Framework; (b) project objective embracing urban services as a whole, while design focused only upon revenue generating municipal markets; (c) precarious commitment of the majority of selected ULGs not perceived by the Bank; (d) project design did not contemplate a straightforward process for replacing withdrawn ULGs since such withdrawals were not recognized as a project risk at appraisal; (e) no economic cost-benefit analysis carried out at appraisal. The flawed design made Bank supervision very time consuming. Supervision missions had to concentrate their work on negotiating and admitting additional ULGs, paying less attention to important project needs such as further revision of the project objectives and design to make them more consistent Borrower Performance : Moderately Moderately The assessment of Implementing Satisfactory Unsatisfactory Agency Performance did not take account of the poor performance of the 8 signed up ULGs that withdrew prematurely from a project, for which they had received Bank approval to participate and to which they were bound by a formal commitment letter. The 10 ULGs that eventually participated in the project were weak, constrained and unready for project implementation. Their performance was mixed. Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: From the ten lessons provided by the ICR, IEG highlights and partly adapts the following:  Lessons learned and extensive experiences gained from previous Bank financed projects about the relationships between central and local governments and the logistics of working across a far-spread country should be key inputs into the design of new operations. In this case, project implementation suffered from protracted bureaucracy at both levels. It also found that working with subnational clients scattered across Indonesia was complex and costly. These real challenges could have been better mitigated had the project design made better use of extensive Bank knowledge of them that already existed.  As the funding of local subprojects through sub-loan agreements can be convoluted and protracted, future projects could consider alternative funding mechanisms. One could be the use of a financial intermediary facility when numerous local governments are involved.  The costs and benefits for a project open to all ULGs compared with one for regional clustering of ULGs should be considered very carefully at appraisal, given the high cost of managing project investments scattered widely across a large country.  Given the inevitable (and often desirable) political flux and administrative restructuring of a major reform drive directly supported by a project itself, its design should have a longer implementation and disbursement timelines than for an operation executed in more stable context. IEG itself adds the following lessons:  The Bank should exercise great care to ensure the consistency of the formulation of the project objective in all project documents, notably the legal agreements, the project appraisal report and the results framework. Even a minor variance of wording, from “improving the provision of urban services” to “improving urban services” in this case, can imply major changes in the design and the direction of a project.  An assessment of efficiency should be a crucial part of evaluating the performance of any investment operation. The lack of an overall assessment of efficiency, as in this case, precludes well based findings about how well a portfolio of investments performed thus far and how well it might perform in the future.  For a good understanding of the potential and actual scope of project results, it is important to incorporate absolute numbers of intended beneficiaries into a project design, and the numbers of actual beneficiaries into its evaluation. This is especially important for the current project that was trying to make ULG administration transparent to more people and encourage greater public participation. In the case of this project, performance indicator targets expressed as a percentage of ULGs were met or exceeded, but the overall number of people benefitting fell significantly because the ULGs that actually participated in the project were smaller than the ULGs originally selected at appraisal. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provides broad coverage of this operation's complex story of implementation and the project's efforts to stay on track. In so doing, the report provides a useful introduction to how it performed, candidly acknowledging key shortcomings. It does this at considerable length with a main text running to 40 pages. In spite of this, there are some omissions: (i) a lack of an economic or financial analysis covering the whole project; (ii) incomplete assessment of the relevance of the project objectives to government priorities and strategies; and (iii) no assessment of the use of US$5.0 million Japanese trust funding of the project (as the task team later informed IEG, this was because the information was not available at the time of the issue of the ICR). Some errors include: (i) incorrectly reporting of PAD’s version of the project objective as being “improving the provision of urban services” (Basic Data Section F. and p.3); (ii) mixing up Government and Implementation Agency roles under the assessment of Borrower Performance; (iii) incorrect reporting of project costs at appraisal (ICR p. 41), omitting the 25% of project costs assigned to contingencies and taxes and duties by the PAD (p. 37). Taking into account its strengths and weaknesses, IEG rates the quality of the ICR, on balance, positively. a.Quality of ICR Rating : Satisfactory