Document of The World Bank Report No.: 77878 PROJECT PERFORMANCE ASSESSMENT REPORT REPUBLIC OF INDONESIA PROVINCIAL HEALTH I (CREDIT NO. 33810) PROVINCIAL HEALTH II (LOAN NO. 46290 AND CREDIT NO. 35370) AND HEALTH WORK FORCE AND SERVICES PROJECT/ PROVINCIAL HEALTH III (LOAN NO. 47020/CREDIT NO. 37840) November 13, 2013 IEG Public Sector Evaluation Independent Evaluation Group ÿþii Currency Equivalents (annual averages) Currency Unit = Indonesian Rupiah (Rp) 2000 US$1.00 Rp 7,468 2005 US$1.00 Rp 9,240 2001 US$1.00 Rp 9,395 2006 US$1.00 Rp 9,330 2002 US$1.00 Rp 10,319 2007 US$1.00 Rp 9,330 2003 US$1.00 Rp 8,876 2008 US$1.00 Rp 9,234 2004 US$1.00 Rp 8,441 Abbreviations and Acronyms ADB Asian Development Bank CAS Country Assistance Strategy DCA Development Credit Agreement DFA District Funding Allocation GDP Gross domestic product HIV Human immunodeficiency Virus HWSP Health Workforce and Services Project ICR Implementation Completion and Results Report IDA International Development Association IDHS Indonesia Demographic and Health Survey IEG Independent Evaluation Group JHC Joint Health Council IMR Infant mortality rate M&E Monitoring and evaluation MMR Maternal mortality rate NGO Nongovernmental Organization PAD Project Appraisal Document PDO Project Development Objective PHP Provincial Health Project PPAR Project Performance Assessment Report SDR Special Drawing Rights TF Task Force TRT Technical Review Team U5MR Under 5 mortality rate USAID United States Agency for International Aid Fiscal Year Government: January 1 - December 31 Director-General, Independent Evaluation Ms. Caroline Heider Manager, IEG Public Sector Evaluation Mr Mark Sundberg Task Manager Ms. Pia Schneider 111 Contents Principal Ratings.................................................... vii Key Staff Responsible...........................vm...... ............viii Preface........................................................... xi Summary ........................................................ xiii 1. Background and Context................................................. 1 Introduction...................................................... 1 Decentralization - Implementation.....................................1 Challenges to Decentralization.........................................2 Decentralization - Impact...........................................4 The Bank Projects..................................................5 Comparative Overview of the Three Projects..............................6 2. Provincial Health Project 1 (PHPl)......................................8 Objectives, Design, and Relevance........................................8 Objectives......................................................8 Relevance of the Objective..........................................8 Design..... ...................................................9 Relevance of Design..............................................11 Monitoring and Evaluation Design....................................12 Implementation...................................................12 Achievement of the Objectives.........................................15 Ratings......................................................... 19 Outcome...................................................... 19 Risk to Development Outcome....................................... 19 Bank Performance............................................... 20 Borrower Performance............................................ 21 Monitoring and Evaluation........................................... 22 3. Provincial Health Project 2 (PHP2)..................................... 23 Objectives, Design, and Relevance...................................... 23 Objectives..................................................... 23 Relevance of the Objective......................................... 23 This report was prepared by Hjalte Sederlof who assessed the project in November 2012, together with Moritz Piatti. The report was peer reviewed by Bjorn Olof Ekman and panel reviewed by Robert Mark Lacey. Viktoriya Yevsyeyeva provided administrative support. iv Design ........................................................ 24 Relevance of Design ........................ ................ 26 Monitoring and Evaluation ............................................ 28 Achievement of the Objectives ........................................ 29 Efficiency ....................................................... 33 Ratings ......................................................... 33 Outcome ...................................................... 33 Risk to Development Outcome ....................................... 34 Bank Performance......................... ................ 34 Borrower Performance ..................................... ....... 36 Implementing Agency Performance ................................... 36 Monitoring and Evaluation .......................................... 37 4. Health Workforce and Services Project (Provincial Health Project 3) - HWS................ 37 Objectives, Design, and Relevance............................... ........ 37 Objective ...................................................... 37 Relevance of the Objective .................................. ....... 38 Design ........................................................ 38 Relevance of Design ........................ ................ 40 Monitoring and Evaluation Design ............................. ....... 41 Implementation ................................................... 41 Achievement of Objectives ........................................... 43 Efficiency ....................................................... 46 Ratings ......................................................... 47 Outcome ...................................................... 47 Bank Performance................................................ 48 Borrower Performance .................................... ........ 49 Implementing Agency Performance ................................... 49 Monitoring and Evaluation .................................. ........ 50 Summary Conclusions from the Three Projects ............................. 50 Critical Assumptions............................................... 50 Did the Projects Make a Measurable Difference? ................ .... ....... 51 5. Lessons......................................................... 56 References ........................................................ 59 Annex A. Basic Data Sheet for Indonesia Provincial Health I Project Name (P049545)...... 63 V Annex B. Basic Data Sheet for Indonesia Provincial Health II Project (P049539)............ 66 Annex C. Basic Data Sheet for Indonesia Health Workforce and Service Project (P073772)... .............................................................................................................6 9 Annex D. Results of Workshops on Lessons Learned from the Decentralization Projects.... 72 Annex E. List of Persons Met ............................ ...... ......... 73 Tables Table 1.1. Indonesia: Change in Key Health Indicators 1992-1997 and 2002-2007............ 5 Table 1.2. Provincial Health Project I...................................... 6 Table 1.3. Health Workforce and Services Project/Provincial Health III... ................... 6 Table 1.4. Provincial Health Project II....................7..... .............7 Table 2.1. Provincial Health Project 1 (PHP1) Key Dates .................................................. 13 Table 2.2. Provincial Health Project 1 (PHPl) Project Cost and Financing........................ 13 Table 2.3. Provincial Health Project 1 (PHPl) Financing.................................................. 13 Table 3.1. Provincial Health Project 2 (PHP2) Key Dates .................................................. 27 Table 3.2. Provincial Health Project 2 (PHP2) Project Costs by Component.................... 27 Table 3.3. Provincial Health Project 2 (PHP2) Financing .................................................. 27 Table 4.1. Health Workforce and Services Project (Provincial Health Project 3) Key Dates.... .............................................................................................................4 1 Table 4.2. Health Workforce and Services Project (Provincial Health Project 3) Planned vs. Actual Disbursements ...................... .................. 42 Table 4.3. Health Workforce and Services Project (Provincial Health Project 3) Financing. 42 Table 4.4. Rates of Change (annual percentage rates) in Selected Health Services and Outcome Indicators, 1997-2007............... .................... ...... 54 Figures Figure 4.1. Indonesia - Trends in Infant and Under-5 Mortality Rates, Various Years......... 52 Figure 4.2. Indonesia - Trends in Selected Indicators of Service Provision, Selected Years ...................................................53  'fl. cl. QU) Q cac4 > o o cl - 11 cl.ee e ~ ~~~ u U eee b M e E o 2 o o 2 CL = eCL caý -C2 c E .5 5o a . o o viii Key Staff Responsible Provincial Health I Project Task Manager/Leader Division Chief/ Country Director Sector Director Appraisal Samuel Lieberman Alan Ruby Mark Baird Completion Puti Marzoeki Fadia M. Saadah Joachim von Amsberg Provincial Health II Division Chief/ Project Task Manager/Leader Secto Dieo Country Director Sector Director Appraisal Samuel Lieberman Maureen Law Mark Baird Completion Puti Marzoeki Fadia M. Saadah Joachim von Amsberg Health Workforce and Services/Provincial Health III Project Task Manager/Leader Division Chief/ Country Director Sector Director Appraisal Enis Baris Emmanuel Jimenez Andrew Steer Completion Puti Marzoeki Juan Pablo Uribe Joachim von Amsberg ix IEG Mission: Improving World Bank Group development results through excellence in independent evaluation. About this Report The Independent Evaluation Group assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the Bank's self-evaluation process and to verify that the Bank's work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20-25 percent of the Bank's lending operations through field work. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, and interview Bank staff and other donor agency staff both at headquarters and in local offices as appropriate. Each PPAR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible Bank department. The PPAR is also sent to the borrower for review. IEG incorporates both Bank and borrower comments as appropriate, and the borrowers' comments are attached to the document that is sent to the Bank's Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG's use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://ieg.worldbankgroup.org). Outcome: The extent to which the operation's major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project's objectives are consistent with the country's current development priorities and with current Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, Sector Strategy Papers, Operational Policies). Relevance of design is the extent to which the project's design is consistent with the stated objectives. Efficacy is the extent to which the project's objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. The efficiency dimension generally is not applied to adjustment operations. Possible ratings for Outcome: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for Risk to Development Outcome: High, Significant, Moderate, Negligible to Low, Not Evaluable. Bank Performance: The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan/credit closing, toward the achievement of development outcomes. The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for Borrower Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory.  xi Preface This report assesses the performance of three health projects in Indonesia. It analyzes the context of sectoral decentralization into which the projects were introduced and includes Project Performance Assessment Reports (PPARs) for each project. The three projects were the Provincial Health Project (IDA Credit No. 33810), approved on June 15, 2000, becoming effective on August 21, 2000, and closed December 31, 2007; the Provincial Health Project II (Loan No. 46290 and IDA Credit No. 35370), approved on June 26, 2001, becoming effective on April 24, 2002, and closed July 31, 2009; and the Health Work Force and Services Project/Provincial Health III (Loan No. 47020/Credit No. 37840), approved on June 12, 2003, becoming effective on December 30, 2003, and closed December 31, 2008. The report was prepared by Hjalte Sederlof, IEG consultant with research assistance provided by Moritz Piatti, Junior Professional. It presents findings based on a three-week mission to Indonesia from October 25 to November 21, 2012, as well as a review of project appraisal documents, sector reports, and other relevant materials. A list of people met is in Annex E. The cooperation and assistance of all stakeholders and the support of the World Bank country office in Jakarta are gratefully acknowledged. Following standard IEG procedures, a draft of the report was sent to the Borrower for comments before being finalized. No comments were received.  xiii Summary This Project Performance Assessment Report (PPAR) evaluates three Indonesia health projects initiated over 2000-2003, while the government was decentralizing several sectors, including the health sector. The projects are Provincial Health (P049545; 2000-2006; IDA $31.26 million), Provincial Health 2 (P043539; 2001-2007; IBRD $44.96 million; IDA $12.02 million), and Health Workforce and Services (P073772; 2003-2008; IBRD $57.49 million; IDA $54.78 million). They shared two Development Objectives related to decentralization: piloting effective health sector decentralization in selected provinces and supporting the Ministry of Health (MOH) in its new oversight role in a decentralized health system. The projects aimed to accomplish these objectives by developing leadership and setting standards in the MOH, strengthening technical support and quality control functions in selected provinces, and building local planning and implementation capacity at the district level. The Health Workforce and Services project also had a third objective: to support effective health workforce policy and management. While decentralization, launched in 1999, was primarily driven by political considerations, it was also seen as an opportunity to revitalize a public health system that was facing coverage, quality, and equity challenges, as well as declining demand for public health services. Decentralization, with its emphasis on local resource management and direct accountability to local constituent groups, was thought to offer a means to address these challenges. The Bank supported this approach in its sector work and policy dialogue, and through lending. Initial legislation issued guidelines for assigning functions to the central, provincial, and district levels. In key areas of resource management, local decision making remained limited, constraining one of the main advantages of decentralization - the application of resources in ways that might better respond to local needs. The three projects were introduced into this decentralizing environment, and the design of all three reflected assumptions about decentralization: that there was agreement within the government on the future roles and responsibilities of the three government levels, and that these were understood by all parties. The Bank recognized the potential risk of proceeding without clear government guidelines, as the institution and capacity building under the projects might turn out to be of only limited relevance. But the risk was considered worth taking. The projects did not achieve their objectives. Decentralization did not proceed as had been expected: a large measure of authority was retained at the center; provinces essentially became extensions of the central ministry; and while districts did "cut loose" from the center as accountability refocused on district authorities and local populations, their control over resources was circumscribed. The following challenges arose for all three projects: * Their designs assumed roles and responsibilities that had not been sufficiently defined for different government levels, and many of the initiatives launched under the projects would turn out not to be sustainable. Xiv * Results frameworks were deficient. They did not succeed in linking process-oriented output indicators to outcomes, and baselines and targets were only identified late in project implementation. * Monitoring and evaluation design initially lacked measurable indicators to track key links in the results chain. This reflected weaknesses in the results chain as well as a lack of monitoring and evaluation planning. The outcome of all three projects is rated unsatisfactory. The relevance of their development objectives is rated high and the relevance of project design is rated modest. All three addressed the key policy variable of decentralization, but they did so with designs for decentralization that the government was not ready for and that exceeded the absorptive capacity of the clients. For PHP1, achievement of the two development objectives is rated modest and negligible; for PHP2 and HWS, achievement of both objectives is rated modest. In each case, the ratings reflected a low rate of outcome achievement. The efficiency of all three projects is rated negligible, as they fell short of achieving their efficiency targets. Moreover, parts of PHP2 were cancelled before completion due to fraudulent financial practices. Bank performance is rated unsatisfactory for all three projects, mainly for lack of adaptation to the decentralization circumstances as they unfolded. Borrower performance also is rated unsatisfactory in all three projects, reflecting the absence of progress on decentralization in all three cases, and recurrent implementation issues. The main lessons to draw from the three projects are the following: * Recognize and plan for uncertainties of change. When undertaking extensive policy reforms in a rapidly changing political and institutional environment, challenges and risks posed by traditional ways of doing business are likely to be formidable. They often are of a political economy nature, as well as technical; and both kinds of risks need to be recognized and planned for. In Indonesia, reluctance to introduce accompanying regulations to new laws was an indicator that new institutional arrangements had yet to mature; and that government was not yet ready to firm up its options for defining new responsibilities. * Monitoring and evaluation. High-risk/high potential projects can be supported, but a prerequisite must be that strong M&E is in place, and that it incorporates appropriate technical as well as political economy indicators; includes a schedule for rigorous assessment of progress and results; and allows lessons to be drawn. * Goals and project organization need to be well understood and supported by the client for effective M&E and learning. Successful introduction of new institutions requires that clients - in this case MOH and sub-national governments - understand the changes that are being sought, believe they are feasible, support them and are ready to pursue them. This requires collaborative project development and continuous socialization of the project among participants to ensure that the client understands what is going on, supports it, and ultimately owns it. * An objective assessment of the client's technical capabilities to take on new roles is a prerequisite for success. The design and timing of inputs - the roadmap for implementation - must be calibrated to client absorptive capacity, especially where there xv may be multiple clients of varying levels of maturity. In this case capacity was strong in some localities, but limitation in others made project targets far less feasible. * Ensure institutional flexibility. Conventional government processes (budget processes, fiduciary controls) should be amenable to adjustment, especially when the emphasis is on piloting new methods. The purpose of piloting is to learn and adapt, which requires some flexibility. * Sequencing of projects. All three projects were piloting new complex reforms in an unsettled environment. All three were closely sequenced, offering limited opportunity for transmitting learning from one project to another. Projects introducing new institutions must be seen as opportunities for learning and improving, and sequencing should have that in mind. Caroline Heider Director-General Evaluation  1 1. Background and Context Introduction 1.1 This Project Performance Assessment Report (PPAR) reviews World Bank support for decentralization of the health sector in Indonesia over the period 2000 to 2008, as provided by three investment projects: Provincial Health 1 (PHP 1), Provincial Health 2 (PHP2), and Health Work Force and Services (HWS). 1.2 The three projects were designed to pilot effective health sector decentralization in Indonesia. They coincided with the introduction of decentralization legislation in 1999, and focused on building the capacity of provincial and district health authorities to plan, implement, and evaluate programs as well as the capacity of the central Ministry of Health (MOH) to support province and district efforts. While legislation primarily was driven by political considerations of national unity, the health sector was at the forefront of the changes that were taking place in the organization of government. This meant that the health system would be shifting from a strongly centralized one where the MOH functioned both as system manager and service provider, to one where much of planning and management of resource use, and service provision, would be devolved to district authorities, supported by provincial authorities. 1.3 The health community and donors, including the World Bank, saw decentralization as an opportunity to address some of the challenges that had arisen over time in the health sector and were influencing its performance: access to services was limited and inequities were large - across regions, between rural and urban areas, and between rich and poor; service quality was low, reflecting low levels of spending, weak management, and an insufficient and insufficiently prepared and often unmotivated workforce. Low motivation was exacerbated by a rigidly hierarchical command and control system applied by the MOH, which limited discretionary decision making by district staff or health workers (Heywood and others 2009). Demand for public health services had reached a low point of less than 20 percent in 1998, as clients turned to a growing cadre of private practitioners or to self- medication. While health outcomes had been steadily improving, comparable countries were showing better results: by 1997 infant and maternal mortality rates were 43 per 1,000 and 334 per 100,000 in Indonesia, compared to 30 and 160 in Vietnam, and 33 and 140 in the Philippines (IDHS 1997). DECENTRALIZATION - IMPLEMENTATION 1.4 In 1999, the government launched what became known as the "Big Bang" approach to decentralization, describing its almost immediate implementation. Two laws on regional autonomy were passed, one devolved most functions of government to the provinces and districts, and the other altered the financial balance between the center and provinces and districts.' A deadline for implementation was set for January 1, 2001. In the health sector, the 1 Decentralization was launched with the enactment of Law No. 22/1999 or regional administration and Law No. 25/1999 on fiscal balance. They were complemented by Law No. 34/2000 on regional taxation; and government decrees in 2004 and 2007. 2 legislation redefined relationships between the central MOH, provinces and districts, albeit without specifying a detailed distribution of functions. The district health authority would become pivotal to local-level health policy formulation, planning, and provision of primary care, including district-level hospital care. Existing service networks were folded into local government structures along with a quarter of a million health workers and substantial financial resources. Only tertiary and specialized care remained outside district control, managed by the central ministry. Accountability arrangements changed, as district health authorities no longer reported to the central MOH but to district political authorities and local civil society. The central MOH was to exercise sector leadership, shape national policy and determine national priorities, set standards, and manage tertiary and specialized services and vertical programs. Provinces were expected to provide training, assist districts by coordinating activities that were likely to benefit from economies of scale and externalities across jurisdictions, and support districts that were unable to perform their functions. The government would manifest a great deal of uncertainty with regard to the role of provinces, even to the extent of raising questions about their continued relevance. In practice, they would come to function primarily as an extension of central MOH authority; although with time they also have built up their coordinating function and introduced reforms, including in particular elements of province-level health insurance. CHALLENGES TO DECENTRALIZATION 1.5 The quick launch of decentralization would face a number of challenges with potentially adverse effects on the aims of decentralization. These challenges included unclear roles between government levels, limited management capacity at all levels, constraints on resource management, and reduced information flows for policy making. 1.6 Intergovernmental responsibilities remained unclear. While relationships between levels of government had been defined, decentralization legislation had provided only broad guidelines for assigning functions to the three levels of government; detailed regulations on roles and responsibilities were to be issued later.2 This uncertain regulatory context, which remains to this day, may have reflected uncertainty about the exact shape a decentralized health sector should take, and what responsibilities were appropriate at which level of government. Insufficient clarity of the role of provinces, concerns about management capacity at district levels, which varied greatly between districts, concerns about decentralization in MOH, and the effects of entrenched interests in different functional areas, all may have contributed to the situation. Subsequent regulations would attempt to clarify assignments but would prove insufficient, and there remains to this day a need to better determine functional responsibilities.3 In the meantime, districts have gone ahead and adapted to the situation, defining responsibilities as a function of available capacities. The 2 Functions include financing, policy planning and budgeting, service delivery including procurement and construction; human resource management including education, training, recruitment, remuneration and deployment. 3 This subsequent legislation included a decree issued in 2007, which noted functions that were to be shared, including the ones listed in the footnote above, but not how they were to be shared between government levels. 3 relative vacuum did have adverse overall effects on the system: (i) accountability would prove hard to pinpoint, as districts found it relatively easy to blame the central MOH for problems at the district level; and (ii) considerable overlap and duplication of tasks appeared across levels of government, generating inefficiencies in the organization and delivery of services (World Bank 2005). 1.7 Districts' control over financial resources was limited, and there were inefficiencies in the flow of funds. Significant constraints remained for district health authorities in the use of financial resources. While central government transfers to districts increased and districts were given access to own-source revenues, over half of all moneys were earmarked for staff salaries and MOH-approved investments in infrastructure and equipment, and to meet matching obligations. Moreover, transfers fluctuated from one year to the next, and transfers were often not timely. This had the effect of disrupting district budgeting and spending plans, often resulting in under-spending. 1.8 A persistent concern, even before decentralization, had been the relatively low overall level of public funding for health, and it was often proffered as the main reason for the disappointing performance in the sector. It was hoped that increased transfers and the introduction of own-source revenues would resolve the issue. And, in fact, district health budgets tripled over the period 2001 to 2007 (NHA 2005; World Bank 2008).4 This did not, however, prove sufficient: a recent study (ADB 2011) points to persistent vertical and horizontal imbalances, as resources were both insufficient and badly allocated. Possible explanations for both imbalances can be found. The primary health system was expensive and it was rapidly expanding; national service standards (inputs of staff and equipment) were set high; and curative care absorbed some 40 percent of moneys. Horizontal equity was influenced by allocation formulas that were not sufficiently poverty calibrated; and weak capacity to generate own resources in poor districts. 1.9 Human resource management remained largely centralized and lacked the necessary information base for effective management. While civil servants reported to district authorities on their day-to-day activities, they were paid, hired and fired, and assigned to districts by the central government. Once assigned, district authorities could allocate them according to need. Districts could hire (temporary) contract staff using their own resources, but with time this would turn out to be a limited option, as most own resources already were consumed by non-salary operating costs and matching federal transfers. Human resource management was further complicated by the absence of a coherent information system at either national or sub-national levels. Information on stocks and flows was weak in all phases of personnel management, from entry through active workforce stages (supply and distribution of workforce, including measures of capacity, motivation, and performance) to attrition. In the absence of information, systematic planning was difficult, and staffing requests from districts to the center would be ad hoc, based on urgent needs or centrally issued staffing guidelines (informal communication). 4 This tripling of resources was partly driven by the introduction of a poverty-targeted budget- financed health assistance scheme in 2005. 4 1.10 Managerial and technical capacities to take on new responsibilities were weak at all levels. None of the three government levels were well-prepared for taking on their new responsibilities. District governments had limited experience in managing their own affairs after years of central authority. They had neither the scale, depth, numbers of staff nor other resources to carry out all the health responsibilities that were being transferred to them. The MOH was hardly better placed: staff were not well prepared to take on many of the tasks envisaged for the ministry: it did not have the necessary skills and experience to assume a new role as system manager rather than main provider, and with districts reporting on a voluntary basis, it no longer had the kind of system oversight that would have supported easy transformation into its new role. An assessment undertaken in 2010 notes the continued absence of leadership and vision at all levels in the sector, and a tendency to try to implement the old system a little bit more in responding to health problems (Heywood and others, 2010). Another study in 2011 points to the need for better financial management to strengthen planning, implementation, and monitoring at the district level (Private communication, 2011). 1.11 Information systems were inadequate. Decentralization accentuated existing weaknesses in health information systems. When reporting obligations of the district to MOH became voluntary, information flows on service statistics and outcomes, which already were of incomplete and infrequent, would make system oversight and national planning even more difficult. The information system for human resource statistics, which included a series of incompletely connected systems for collection and distribution, would weaken already weak human resource management and lead to the kind of ad hoc situations described in the section on human resources. 1.12 In summary, the challenges that the health sector faced with decentralization- unclear mandates, insufficient skills, limited flexibility in the use of resources, and uncertain information flows-could be expected to have a strong adverse effect on local decision making and ultimately on outcomes. DECENTRALIZATION - IMPACT 1.13 While attributing any subsequent changes in health system performance to decentralization is difficult without a counterfactual (decentralization was introduced at the same time nationwide), comparing trends pre- and post-decentralization using key outcome indicators can serve as a rough proxy for the decentralization effect. Maternal, infant, and under-five mortality can therefore be used as key indicators to examine the periods 1992- 1997 and 2002-2007 (see Table 1.1). All three indicators show a slowdown in progress. The change is strongest in under-five mortality, where the rate slowed from 67 percent over the period 1992-1997 to 2 percent over 2002-2007. Observers (informal interviews) believe that the challenges to the system listed above, and especially the constraints on resource use (staffing and finance), were significant in contributing to the modest performance. Arguably, the slowdown in 2002-2007 could be attributed to high levels of performance already having been achieved, but this is not the case, as the country's indicators were still lagging significantly behind Millennium Development Goal targets. 5 Table 1.1. Indonesia: Change in Key Health Indicators 1992-1997 and 2002-2007 Indicators 1992 1997 2002 2007 Change % 92-97 02-07 MMR/100.000 425 334 307 228 27 26 IMR/1.000 74 52 43 39 42 10 U5MR/1.000 97 58 46 45 67 2 Source: DHS 2003, 2007; BPS (various years) The Bank Projects 1.14 The three projects reviewed were introduced into this context. They were launched at the very beginning of decentralization and were designed as pilot schemes for building capacity at the province and district levels, which would support the transfer of resources and responsibilities to them. They also included capacity building for the MOH to strengthen its role in shaping national policy and supporting decentralization. The projects built on a long- term dialogue between the government and the Bank to improve the effectiveness of primary care, including a larger role for local authorities in setting priorities and designing health initiatives. A number of projects throughout the 1990s had begun to build capacity at the province and district level that went beyond skills for service delivery to achieve effective local health interventions, introducing concepts of management, budgeting, and planning. This was particularly the case with the Third Health Project that closed in 1996, and the Third Community Health and Nutrition Project that closed in 2001 . Their successful implementation was one encouraging factor behind the designs of the three decentralization projects. However, decentralization would not take the path that the Bank assumed in preparing the three projects reducing the effectiveness of processes and skills that were being transmitted through the projects. This is starkly illustrated by a large grants component supporting implementation of the three projects, of which only some 74 percent was disbursed. These issues are reflected in the assessment below of the three projects. ' Other relevant projects included the Fourth Health Project (Loan 3905-IND), the Fifth Health Project (Loan 4374-IND) and the Safe Motherhood Project (Loan 4207-IND). 6 COMPARATIVE OVERVIEW OF THE THREE PROJECTS Table 1.2. Provincial Health Project I PDO * To bringing about effective health sector decentralization in Lampung and Yogyakarta * To help the central health ministry carry out its new role in a decentralized system Main indicators Achievement of Indicators * Introduction and implementation of new . Duplicate district functions eliminated organization structures, health financing * Health information system introduced in one mechanisms, and work force policies province * Competence testing of para-medical staff and primary health care (PHC) facility accreditation in Yogyakarta * Public accountability mechanisms introduced in Yogyakarta * Establishment and operation of effective, * Health service use increased among the poor locally specific curative service delivery and non-poor * Establishment and operation of effective, * No measurable results locally specific public health service provision * No measurable results and financing arrangements Key ratinRs Outcome: Risk to development Bank performance: Borrower performance: Unsatisfactory outcome: Significant Unsatisfactory Unsatisfactory Source: PHP1 Table 1.3. Health Workforce and Services Project/Provincial Health III PDO To achieve the following in a decentralized system: * An effective and equitable delivery of health services of enhanced quality and better health outcomes in the districts of Jambi, East Kalimantan, West Kalimantan and West Sumatra * Effective health workforce policy management and development * Effective stewardship of the health sector Main indicators Achievement of Indicators * Decrease in under-5 MR to 35 per 1000 live births * Positive trend, but target rate was not achieved * 90 percent of births attended by a skilled birth attendant * Positive trend, but target rate was not achieved * 90 percent of pregnant women have at least 4 ante-natal * Positive trend, but target rate was not achieved visits * Service use by the poorest quintile (Q1) at least 50 * Positive trend, but target rate was not achieved percent * Practically no increases; target increases were * The share of health spending in total local government not achieved spending at 15 percent; 8 percent in East Kalimantan * National medical examination and certification * Institutional structure and instruments in place for a system developed national medical examination and certification * No baseline; target was not achieved * At least 80 percent of medical students in two selected universities satisfied with curriculum and education provided Key ratings Outcome: Risk to development Bank performance: Borrower Unsatisfactory outcome: Significant Unsatisfactory performance: Unsatisfactory Source: HWS 7 Table 1.4. Provincial Health Project II PDO * To bring about effective health sector decentralization in Banten, North Sumatra and West Jawa * To help the Ministry of Health and Social Welfare carry out its roles in a decentralized system Main indicators Achievement ofIndicators * Introduction and implementation of new organization * New personnel policies: no results structures, health financing mechanisms, and work * Reduction in non-health staff: no results force policies * Province specific health information systems: no results s Licensing and accreditation introduced in North Sumatra, West Jawa * Accountability mechanisms: target not achieved h Hospitals, health centers become autonomous units: target irrelevant with new legislation * Health insurance coverage: no target set, but coverage increased in all provinces by 1 to 4 percentage points i Use of health services by poor(Q) increased in all provinces by 4 to 13 percentage points * Health promotion: target of 5 proposals achieved in North Sumatra * Health information system introduced in one province Health service use increased among the poor and non-poor * Implementation of effective, locally relevant curative No measurable results * Decentrizadaeiation troducesh d in service delivery and financing arrangementscntalit ni ta e n * Implementation of effective, locally grounded public Ministry of Health health service provision and financing arrangements * National health grants program piloted: reutnotousvaniat:ed;gdtscontivantdwit * Increased awareness in Government and among the Mists nof ealtd icniudb public of health issues/nsks/solutions ry * Indications of enhanced commitment at the province * National health grants program piloted: and district levels including financing of priority results not evaluated; discontinued by programs, and basic services/drugs for the poor, and Ministry of Health inclusion of health indicators in the General Allocation Geohanhic information system formula introduced * Effective partnership arrangements with districts, *Public health laboratory established provinces and NGOs Established * Enhanced effectiveness of communicable disease surveillance * Enhanced food and drug inspection systems and public accountability Key ratings Outcome: Risk to development Bank Borrower Unsatisfactory outcome: Significant performance: performance: Unsatisfactory Unsatisfactory Source. PHP2 8 2. Provincial Health Project 1 (PHPI) Objectives, Design, and Relevance OBJECTIVES 2.1 According to the Development Credit Agreement (DCA), the project objective was to "assist the Borrower in bringing about effective health sector decentralization in Lampung and Yogyakarta and at the national level." 2.2 The Project Appraisal Document (PAD) has two project development objectives. It also has two versions of these objectives, one in the text (page 2) and one in Annex 1, the Project Design Summary. They are substantively similar. 2.3 The project development objectives in the text of the PAD are: "to bring about effective health sector decentralization in the provinces of Lampung and Yogyakarta. The challenge during this period of institutional change and economic recovery will be to protect health services which are essential for the poor and the public at large while initiating key sector reforms and putting health financing on a firm footing"; "to help the central health ministry carry out its new role in a decentralized system in Lampung, Yogyakarta and other provinces. This role encompasses analysis of key issues, advocacy of best practices and standards, and support for local initiatives and innovation. 2.4 The statements of development objectives in the DCA and the PAD are consistent, although not identical. The review will assess the project based on the objective statement in the PAD, which provides better guidance to "effective decentralization" than does the DCA version of the project development objective. RELEVANCE OF THE OBJECTIVE 2.5 The relevance of the objective is rated high. It directly contributed to government and Country Assistance Strategy (CAS) objectives of effective decentralization. Effective decentralization was a key element in the country's long-term health strategy, and it would remain a primary initiative in the health sector for much of the decade following its introduction in 1999. Here, the project provided a roadmap for functional decentralization and for addressing related human development needs. It supported two of the four pillars in the World Bank's Country Assistance Strategy for Indonesia, covering 2005-2011, including in particular pillars relating to better social services delivery to the poor and to governance and implementation of the decentralization policy. 9 DESIGN 2.6 The project was designed as a pilot to introduce institutional and policy reforms that would facilitate decentralization in the health sector. It included two provinces - Lampung and Yogyakarta - and the central MOH. The design was based on specific roles for the three government levels: for the MOH, leadership and advocacy, information gathering and analysis, and standard setting; for the provinces, technical support and quality control for district programs, communicable disease surveillance and control, health promotion, a larger role in procurement, regulatory tasks, and personnel management; and for districts, local- level planning of health programs and health care provision. Emphasis was placed on building capacity and then using that capacity to build up local health services, with a particular focus on ensuring access to care for the poor. 2.7 The project was to proceed in two phases: an initial one of 18 months consisting of capacity building, technical support, and quality assurance activities. A second phase would fund district health improvement and reform proposals developed by applying the skills learned during the first phase. A number of entities were set up to develop institutions and build up district level capacity; and vet grant proposals and determine health policy at the province level. At the central level, a unit for policy analysis and cross-ministry task forces for decentralization were established; in the province, a Joint Health Council (JHC) and Technical Review Teams (TRT), and at the district level, Task Forces (TF). The Task Forces would support institution building at the district level and help develop medium-term and annual investment plans, and proposals for financing through the grant mechanism. The JHC and the TRT would vet and approve such proposals. It was also envisaged that the JHC gradually would take on a broader oversight and coordinating role. 2.8 Project components as summarized from the Project description in the PAD were as follows: Component A.l: Managing Decentralization (District and Provincial) Estimated cost at appraisal $3.81 million; actual cost $4.57 million 2.9 Project-supported Task Forces were to focus on critical institutional issues and develop recommendations for implementation in Phase 2. Results expected by the end of 2001 include finalization of the division of responsibilities and institutional arrangements between districts and provinces; completion of the merger of the Kanwil and Dinas Kesehatan;6 a human resource development plan that covered possible downsizing, contract hiring and career development; training for management information system staff; baseline surveys, and mapping of poverty areas; and a health promotion strategy including setting up a provincial health promotion board in Yogyakarta. Component A.2: Mobilizing Resources (District and Provincial) Estimated cost at appraisal $1.73 million; actual cost $1.42 million 6 The two are similar administrative health entities in districts with some duplication of functions. 10 2.10 During the first 18 months, Task Forces would assist the provinces in developing integrated health planning and budgeting systems, reviewing and possibly piloting alternative insurance approaches, developing autonomous facilities that rely on cost recovery, and examining tax-based and other means of generating resources for health. The objective was to improve the sustainability of health services through reforms in budgeting and resource mobilization mechanisms. Component A.3: Improving Health Service Access and Quality (District and Provincial) Estimated cost at appraisal $4.17 million; actual cost $1.90 million 2.11 During 2000 and 2001, this component would supplement current government outlays on essential health services by financing needed inputs, improved access to facilities, and specific services. In addition, funds would be allocated to support introduction of quality assurance programs and intensified health promotion. Component A.4: Grants Estimated cost at appraisal $63.71 million; actual cost $99.27 million 2.12 Starting in 2002, funds would be made available to districts through three different grant mechanisms. Two would finance implementation of recommendations developed by the Task Forces (components Al and A2) operating in Phase 1. The third grant mechanism would finance reform and improvement of district-level health services. Districts would prepare yearly proposals to be evaluated by the Technical Review Team and the Joint Health Council at the province level. Proposals would be based on health needs and capacities, and focus on improving coverage and quality of public health services for the entire population, and curative care for the poor. Proposal preparation would be based on the number of poor people in the district, service use by the poor, and spending targets set to attain goals articulated in provincial vision statements, i.e., "Healthy Yogyakarta 2005" and "Healthy Lampung 2010." Funds would be released after proposals had been approved by the Joint Health Council and after being reviewed and approved by IDA. As IDA financial support gradually was to decline in 2003-2005, district grants would be sustained at 2002 levels using funds drawn from local sources. Component A.5: Project Management (District and Provincial) Estimated cost at appraisal $0.78 million; actual cost $2.00 million 2.13 The project would support various consultant services, office equipment and supplies, workshops, and the training and supervision costs incurred in managing the project at the provincial and district levels. Component B.1: Managing Decentralization (Central) Estimated cost at appraisal $2.87 million; actual cost $1.46 million 2.14 This component would help the central Ministry of Health become an effective analytical, advisory, and advocacy agency. The initiatives to be supported included cross- ministry task forces on decentralization and accountability; piloting early retirement and re- deployment measures to facilitate reshaping the health workforce; and social marketing and other mean of highlighting health sector needs. The component would support policy studies 11 commissioned by the new Policy Analysis Unit within the Ministry of Health. This unit would solicit proposals from academic or research institutions, NGOs, and professional organizations. Findings would be made available to all provinces and districts. Finally, this component would fund assessments of project impact. These systematic evaluation studies would be carried out under an independent research program housed outside the Ministry of Health. Component B.2: Resource Mobilization (Central) Estimated cost at appraisal $0.56 million; actual cost $0.13 million 2.15 The Ministry of Health would provide grants to provinces or districts to test resource generation instruments and to fund proposals pertaining to the development of decentralized health financing systems. Proposals would be solicited from provinces and districts outside Yogyakarta and Lampung. Grant mechanisms would be piloted, and approved proposals funded jointly by the central ministry and the regions. Aside from improving local capacity to develop specific programs, it was also envisaged that matching grants could become an instrument which the central ministry could use to encourage local governments to support public health and poverty-oriented activities. Component B.3: Project Management (Central) Estimated cost at appraisal $1.00 million; actual cost $2.29 million 2.16 The project would support various consultant services, office equipment and supplies, training, workshops, and supervision for project management. RELEVANCE OF DESIGN 2.17 Relevance of design is rated modest. The project design was relevant to the development objective. It was straightforward: it addressed institutional barriers and capacity constraints to undertaking the roles defined for the three government levels; and it piloted the new institutions through grants. But the design was not simple, and some of its initiatives would prove to be premature. Provinces and districts had had little practice in decision making prior to decentralization, where a hierarchical, centrally managed and administered command-and-control environment prevailed and where reporting and accountability was bottom up. Now extensive new responsibilities were envisaged under the project, while at the same time uncertainty surrounded the form decentralization might take: while project design reflected international concepts of good practice in decentralization, and had been the focus of extensive dialogue between the Bank and health authorities through much of the 1990s, the regulatory basis for sharing authority between different levels of government was still unclear and would remain so throughout the life of the project and thereafter - human resource management remained centralized; and the use of financial resources constrained. These were challenges that the Bank team may not have sufficiently appreciated at the time of project design. The PAD did rate the overall project risk as "substantial," but does not specifically include any of these as risks. In particular, the definition of a strong intermediary role for the provinces, whatever its merit, was something that the government had not sufficiently considered at the time and that in fact subsequently has been put aside as the structures of decentralization have taken better form. Other concerns in design lay with the 12 arrangements for project management and the results framework, discussed in the sections on Bank and implementing agency performance. 2.18 The results framework as presented in the PAD was insufficient: while it contained the elements of a concise summary of project design, it did not explicitly link outputs to the outcome indicators that underpinned the PDO. This, in turn, was likely to make effective monitoring difficult, as the absence of such links would not allow easy measurement of progress against plans and toward objectives. Moreover, outcome indicators were at a level of generality (and would remain so throughout project implementation) that would make it difficult to determine if satisfactory results were being achieved. The results framework also consisted of a long series of outputs with related indicators, many of which were essentially administrative in nature, and in cases where measurement might have been feasible, initially included neither baselines nor targets. The PAD (page 47) indicates that they were introduced during project negotiations, which raises questions about the extent to which the borrower was committed to the results. In order to be effective and have local stakeholder buy-in that encourages its use, the results framework is better developed during project preparation and in close collaboration with the borrower's teams. MONITORING AND EVALUATION DESIGN 2.19 Monitoring and evaluation was an integral part of the project (Project component B. 1.). It was to be implemented by an independent research unit located in the MOH (Project component B. 1). It was to conduct baseline surveys, yearly tracking surveys and final surveys to assess project impact. The project included three key performance indicators. These were outputs that plausibly supported the development objective. They were of the form "introduction and implementation of' structures, mechanisms, policies, and arrangements. Monitoring mechanisms consisted of conventional recording of inputs and tracking of procurement, disbursement and physical deliverables. This was done by Bank teams and by project management units at the central and district levels. Decentralization task forces and the policy analysis in the central ministry, and the monitoring and support agencies established at the sub-national level (JHC, TRT, TF) monitored technical quality, supported by Bank teams. These arrangements were supported by relevant manuals. A health information system was being put into place by the end of 2003. 2.20 While a design that drew on outside consultants (the independent research unit) may have been a practical solution in a situation of limited capacity for monitoring and evaluation in the MOH, such an approach also carries risks. It may not sufficiently underline the importance of M&E for the client, and this may be better done through the establishment of formal lines of authority with clearly defined responsibilities for collecting, analyzing and reporting performance information. This also ensures continuity for the M&E function. Implementation 2.21 Implementation arrangements. Implementation arrangements were designed to mitigate the risks inherent in introducing new concepts into health administrations that had little experience with decentralization. For day-to-day project implementation, implementation units were set up at district and province levels. Joint Health Councils were 13 set up at the province level to determine broad health policy for the province and to award project grants. These district and provincial bodies were supported by Task Forces for institution building and Technical Review Teams in planning and developing grant proposals. At the central level, a Central Coordinating Unit was located in the Secretary General's office. The central Ministry of Health also was strengthened by policy and cross- ministry decentralization teams that were supposed to become permanent features of the ministry structure. Project management manuals and grant operational manuals served to guide project implementation. Table 2.1. Provincial Health Project 1 (PHP1) Key Dates Process Date Process Original date Revised/actual date(s) CONCEPT REVIEW: 08/14/1998 EFFECTIVENESS: 08/21/2000 08/21/2000 APPRAISAL: 02/16/2000 RESTRUCTURING(S): APPROVAL: 06/15/2000 MID-TERM REVIEW: 05/10/2004 CLOSING: 06/30/2006 12/31/2007 Source: ICR for PHP1 Table 2.2. Provincial Health Project 1 (PHPI) Project Cost and Financing Appraisal Actual latest Percentage of Components estimate estimate (US$ million) (US$ million) appraisal A.1. Managing Decentralization (District and 3.81 4.57 120 Provincial) A.2. Mobilizing Resources (District and 1.73 1.42 82 Provincial A.3. Improving Health Services Access & 4.17 1.90 45.6 Quality (District and Provincial) A.4. Grants 63.71 99.27 155.8 A.5. Project Management (District and 0.78 2.00 256 Provincial) B.1. Managing Decentralization (Central) 2.87 1.46 50.9 B.2. Resource Mobilization (Central) 0.56 0.13 23.2 B.3. Project Management (Central) 1.00 2.29 229 Total Project Costs 78.63 113.04 143.8 Source:ICR for PHP1 Table 2.3. Provincial Health Project 1 (PHPI) Financing Appraisal Actual/latest Percentage of Source offunds estimate estimate (US$ million) (US$ million) appraisal Borrower 40.6 34.4 84.7 International Development 38.0 31.26 82.3 Association (IDA) Source: ICR for PHP1 14 2.22 The total project cost was $113 million or 143 percent of original estimate, due largely to the extension of the project, allowing for six instead of four rounds of grant awards to districts, and additional counterpart financing to cover the additional time. Also, the value of the credit (set in SDRs) increased in terms of U.S. dollars because of changes in the value of the dollar during the project life. The implementation period was extended by 18 months. The IDA credit was 76 percent disbursed and 6.80 million SDRs were cancelled. 2.23 Implementation experience. A number of factors influenced the implementation experience, including the relative capacity of the two selected provinces; a cumbersome budgeting process; the introduction of support mechanisms for district planning and investment activities (the JHCs, TRTs and TFs); and the continued absence of regulations on decentralization. First, while Yogyakarta was able to fall back on a strong human resource base from which to draw consultants, essentially Gajah Mada University, such circumstances were absent in Lampung. Lampung had to rely on short-term outside technical assistance, weakening the capacity building effect that might have been gained from more permanent and locally grounded arrangements. Second, the uncertainty surrounding the authority of provinces and districts to take action - notably in resource management - had a negative influence on some of the initiatives envisaged by the project: authorities were reluctant to move ahead, and when they did, measures were not always implemented in a manner that would assure sustainability. Third, a cumbersome budget process, which had not been anticipated during project preparation, led to severe lags in disbursements for pilot investments. Fourth,, the support mechanisms set up under the project at the province level to facilitate project-related district planning and investments played a positive role in facilitating cross-district communications and in promoting, evaluating, and advising on district investments. 2.24 Implementation of monitoring and evaluation was weak. The independent research unit in charge of M&E was not up to the job, and it was replaced by another team towards the end of the project. While much relevant output information was produced, there is almost nothing on outcomes, including in particular the grants program, where results are limited to spending figures. This is surprising, considering that the value of the grants was some 88 percent of total project costs. M&E was also rendered more difficult as new accountability arrangements where district agencies (health and other) primarily reported to district governments, disrupted information flows towards the center. Reporting and monitoring would prove to work better within districts, where it did allow drawing of conclusions and introduction of adjustments to district planning and investment proposal designs as the project was being implemented. Still, no monitoring seems to have taken place regarding outputs or outcomes of the grants program. 2.25 With the devolution of authority to the districts, the role of the central ministry in M&E focused on creating capacity to promote decentralization and monitor health outcomes. The former was a new and unfamiliar role for the ministry, for which it did not have the necessary experience. The monitoring of health outcomes would be rendered more difficult over time as districts in many instances reneged on their reporting requirements to MOH in the absence of well-defined accountability arrangements. 15 2.26 Safeguard compliance. The following safeguard policies were applicable to the project: environmental ones, ones relating to indigenous peoples, and resettlement. Provisions in the Project Management Manual and Grants Operational Manual addressed the safeguard triggers. There, three things were required. First, compensation had to be paid to project affected persons - those displaced from land or whose assets were acquired - according to principles and guidelines acceptable to the Association. Second, if appropriate, an environmental impact assessment would be carried out and mitigation measures designed based on environmental standards acceptable to the Association. Third, isolated vulnerable people, if any, would be involved in the design of district health proposals through a participatory process to ensure that benefits received would be in harmony with their economic, social, and cultural preferences. 2.27 Fiduciary compliance. Private auditors selected by the implementing agency for conducting interim audits in the early years of the project did not perform well. When corruption was uncovered in 2005 involving the Central Project Coordinating Unit for PHP2, the ministry took action, while the Bank proposed an action plan to strengthen financial control of all three PHPs and send a clear message of deterrence to all provincial health project staff, as well as attempting to resolve the specific issues of the second project. The action plan included (i) strengthening of controls by removing financial and procurement management control from the Bureau of Planning to relevant units in the MOH, (ii) recruiting a firm to conduct internal audits of the projects while working to strengthen internal control within the ministry, (iii) expanding investigations to include more samples to determine the full extent of ineligible expenditures, (iv) replacing external auditors, and (v) temporary withholding payment in PHP2 until immediate actions were taken. All actions except for (iii) and (v) were also applied to PHP1 and HWS. Until the end of the project, recruitment of a firm to conduct internal audit has not been completed. 2.28 Occasional financial irregularities occurred in PHP1, especially involving procurement at the district level. In general, findings involved small contracts and did not result in the Bank declaring mis-procurement. District staff was not familiar with Bank rules and procedures and despite efforts to clarify project management manual and increase the frequency of procurement training and supervision, it remains difficult to reconcile Bank auditing and procurement requirements with those of the government. Achievement of the Objectives Objective 1. To Bring about Effective Health sector Decentralization in Lampung and Yogyakarta 2.29 Achievement of the objective is rated modest. 2.30 In support of Objective 1, "effective decentralization," the following outcomes were specified: (i) protecting health services that are essential for the poor and the public at large; (ii) initiating key sector reforms; and (iii) putting health financing on a firm footing. A set of output indicators had been defined to measure movement toward these outcomes, albeit without explicitly linking to the outcomes. In the analysis in the PPAR, the indicators have been notionally ascribed to outcomes. The extent to which Objective I has been achieved is 16 discussed in paragraphs 2.37 to 2.40 following the bulleted list below associating indicators to outcomes. (i) Protecting health services that are essential for the poor and the public at large * Six rounds of district grants for the establishment and operation of effective locally specific curative service delivery, and public health service provision and financing arrangements. Implemented, using evidence-based planning principles. * Contact rate of poor people at the health centers becomes at least two times a year by January 1, 2003. Partly implemented. Because reliable contact rate data was not available, data on utilization of public facilities by the poor was used. Between 2002 and 2006, utilization by the poorest quintile increased from 33.5 percent to 36.4 percent in Yogyakarta and from 37.9 percent to 40.6 percent in Lampung. Utilization by the near poor (second quintile households) increased from 30.6 percent to 36.4 percent in Yogyakarta and from 27.9 percent to 32.9 percent in Lampung. No targets had been set. (ii) Initiating key sector reforms * The merging of district level Kanwil and Dinas health offices; Joint Health Councils and Technical Review Teams set up and operating at the province level; and Task Forces set up and operating at district levels to strengthen policy and investment planning and implementation Implemented. * Implementation of new personnel policies [at central, province, and district levels] related to recruitment, redeployment, retraining, early retirement, and career development by January 1, 2002. Not implemented - centralized personnel policies maintained * Reduction in the number of nonprofessional staff at the provincial and district health offices by 20 percent in Yogyakarta by January 1, 2004 and in Lampung by December 31, 2005. Partly implemented - nonprofessional staff reduced by 28 percent in Yogyakarta and increased by 65 percent in Lampung. * Implementation of a decentralized and province specific health information system by January 1, 2002. Partly implemented. Yogyakarta has a decentralized information system, adopted by all districts, albeit at a facility coverage of less than 10 percent. Lampung was piloting the Yogyakarta system in one province at the end of the project. 17 * Implementation of new licensing and accreditation system by January 1, 2002. Partly implemented. Yogyakarta established an independent quality control mechanism in 2003 and implemented competence testing for midwives and nurses in 2006. The quality control mechanism received national accreditation in 2007. Lampung conducted a practice assessment in 2003. * Implementation of at least two public accountability arrangements by January 1, 2002. Partly implemented. Yogyakarta established a functioning public accountability unit, an NGO forum, a hotline service to receive complaints, and produced a health newsletter. Lampung conducted a study on public accountability in 2003, but subsequent interventions were limited to the posting of suggestion boxes. (iii) Putting health financing on a firm footing * At least 80 percent of hospitals and 40 percent of health centers were to become semi- autonomous (partially self-financing) units by December 31, 2005. Partly implemented. By project closing, all district hospitals in Yogyakarta had become semi-autonomous; and three of five districts had at least one semi- autonomous health center (less than 40 percent of health centers in the province). In Lampung, 25 percent of district hospitals had become semi-autonomous; three districts had at least one semi-autonomous health center. * At least 50 percent of the province populations are covered by health insurance mechanisms or managed care by December 31, 2005. Partly implemented. In Yogyakarta, insurance coverage had increased from 34.9 percent to 42 percent; in Lampung, it had fallen from 24.5 percent to 22.6 percent. * Six rounds of district grants served to supplement district spending on essential services. Implemented. 2.31 In general, the indicators are not sufficiently coherent to allow determination of whether each of the outcomes was achieved and thus the extent to which Objective I was achieved. At best, district grants (supplementing district spending) should have had an important effect on objectives (i) and (iii), at least during project implementation - but there is no assessment available on the impact the grants really had; and the subnational support mechanisms - JHCs, TRTs, and TFs - built capacity for planning and investment. Available assessments offer a mixed picture of their impact: in some instances, usually where capacity already existed, they brought focus to the development process; in others, their success appears more ambiguous. Typically, they have subsisted in strong provinces, such as Yogyakarta, but either faded or were integrated into the district bureaucracy in other instances. In most instances, outputs appear to have been only partly achieved, at best. In 18 others, evolution of the system has reduced their relevance (personnel policies, hospital and health center autonomy). 2.32 Protecting health services. Of the two indicators grouped under outcome (i), the absence of information on actual spending and outcomes on the first one, on district grants, voids its value as an efficacy measure. The second indicator, on utilization rates, not only reflects positive fallout from decentralization, the higher utilization numbers are also likely to have been influenced by the introduction of a public assistance scheme for health, unrelated to the project. 2.33 Initiating key sector reforms. The sub-national support mechanisms were useful in strengthening and coordinating policy, planning and investment functions at province and district levels. They facilitated project implementation, and they introduced practices that could promote better decision-making at the sub-national level in the future. There is, however, no basis to determine how successful they were, especially in the absence of information about the grant cycle - from decision-making through implementation to results. While the support mechanisms themselves would disappear over time, some of the practices would be incorporated into sub-national decision-making, largely as a function of the capacity of participating provinces and districts to do so. Yogyakarta in particular was able to draw on them in developing its health system, Lampung less so. Capacity is also reflected in the results for other initiatives under this heading: Yogyakarta was able to put into place functioning information systems and quality control mechanisms, and this was not the case in Lampung. 2.34 Health financing. There is almost no information about the grants. They were a large component of the project, and they were to play a key role in inculcating new practices that were being transmitted through the capacity building. Insurance coverage rose modestly in Yogyakarta over the period, and fell in Lampung for reasons that have not been clarified. Objective 2. To Help the Central Health Ministry Carry out its New Role in a Decentralized System 2.35 Efficacy is rated negligible. 2.36 The following two outputs were defined to allow the MOH to carry out its new roles of leadership and advocacy, information gathering and analysis, and standard setting in a decentralized health system: (i) the establishment of a policy unit to enhance capacity to assess health policy; and (ii) the establishment of intra-ministerial decentralization teams to support provinces and districts in implementing decentralization. While these entities were established under the project, there is no evidence that they came to play an important role in either analyzing the decentralization process to define needs or in supporting decentralization at province or district levels. Efficiency 2.37 Overall efficiency is rated negligible. The project appraisal included a cost-benefit analysis based on the impact that policy changes and expenditures might have through access 19 to services and quality of service delivery on a number of diseases that were causing high morbidity, disability, and mortality. This was estimated to result in a benefit-cost ratio of 4.1 for Yogyakarta and 3.7 for Lampung. No post-project benefit-cost ratios were estimated in the ICR, nor does the ICR provide any evidence that access to services or service quality increased as a result of the project. While the impact of improvements in organizational and administrative arrangements on performance was not measured, efficiency gains would appear modest: there is no evidence that the policy and decentralization units that were introduced into the central ministry came to play a significant role either in policy planning or decentralization, and they have subsequently been incorporated into other structures in the ministry; the effects of institution-building in the two participating provinces were modest, as indicated in the efficacy analysis (albeit results were achieved in Yogyakarta, but largely due to a strong local supporting environment); and there is no rigorous assessment of how roughly $100 million in grants was spent or what was achieved with that spending. Ratings OUTCOME 2.38 The outcome is rated unsatisfactory. The relevance of the project development objective is rated high. It directly addressed a key policy variable, that is, decentralization, which featured prominently in the country's development strategy and in the Bank's country assistance strategy. Relevance of project design to achieving this objective is rated modest. While it introduced measures that were likely to support the buildup of an operational framework for health sector decentralization, the design anticipated decision making that either would not be forthcoming, or would be so only partially, as described in the section on Relevance of Design. Moreover, the results framework was weak and capacity of the project management unit insufficient. The rating for efficacy is modest and negligible. Institution building objectives were largely achieved in Yogyakarta, but less so in Lampung and hardly at all in the MOH. Weak monitoring and evaluation precluded the tracking of the use and effects of the district grants. Efficiency is rated negligible, reflecting the lack of evidence on the effects of institution building and the absence of results regarding the grants program. 2.39 The outcome rating differs from the ratings in the ICR and ICRR. In both those cases the outcome was rated moderately satisfactory. The unsatisfactory rating in the PPAR is based on the following: (i) the absence of efforts to reconfigure any part of the project to better accord with the circumstances of decentralization; (ii) the organizational structures put in place under the project - the JHCs, TRTs, and TFs - were not sustainable, nor did they serve as models for subsequent expansion; (iii) there is an absence of evidence on the use or the impact of grant funds; (iv) there is no basis to determine the sustainable capacity building effects on districts of the grant programs; (v) modest improvements in key performance indicators cannot unambiguously be assigned to the project, nor do they deviate from longer- term trends; and (vi) the new entities introduced into the MOH did not come to play a significant role either in policy research or in decentralization. RISK TO DEVELOPMENT OUTCOME 2.40 The risk to development outcome is rated significant. 20 2.41 At appraisal, and in the ICR, the risk to the development outcome is rated substantial. This was a fair assessment at the time, and it is also valid with hindsight. The model of decentralization that was relevant and on which the project was based - a balanced set of responsibilities at the different government levels leading to an optimal use of resources - did not materialize. The roles of the main actors, the central MOH, the province, and the district - evolved in other directions. As a result the effects of the institution building in the MOH and the province have been relatively modest (policy and decentralization initiatives in MOH; capacity building for oversight and coordination in provinces). Institution building at the district level has proven more useful as districts have become the drivers of planning and service provision in the public sector. The grant mechanism that was meant to support and encourage good planning and investment did not prove sustainable in the form envisaged under the project. Capacity building and institutional development more generally would be handicapped by central government constraints on resource use and continued uncertainties regarding roles and responsibilities of the various government levels. Had constraints on resource use been relaxed and roles of actors more precisely defined, the operations development outcomes through its capacity building and institutional development might have been greater. BANK PERFORMANCE 2.42 Quality at Entry. Quality at entry was strong in the following areas. The project was strategically relevant - it focused on decentralization - and the approach was to address major challenges to implementing decentralization. It was underpinned by analytical work. Technical design was sound - it focused on building institutions at all three levels, which were based on international best practice. It included financial and economic analysis, attempted to introduce poverty criteria into district funding allocations, and health promotion to influence healthy behaviors among the target population. Institutional innovations were introduced up front to allow for quick project start-up and ample time for adjusting new processes as necessary during implementation. The latter allowed for a project design that otherwise might have been overly ambitious. 2.43 Quality at entry was weak in its assessment of project risk, the absorptive capacity of the client, in monitoring and evaluation, and in administrative arrangements for implementation. Project risk. While project risks were assessed as substantial, the risk assessment did not address the implications that uncertainties about roles and responsibilities of the three government levels in decentralization might have for the success of the project. Absorptive capacity. The same uncertainties would also affect absorptive capacity: all three government levels were already struggling with the often unclear responsibilities imposed by the decentralization law, on top of which the project introduced innovations that still had little formal backing in law or regulation. Monitoring and evaluation. The formulation of the results framework would make it difficult to construct a satisfactory monitoring and evaluation system, and this would make it difficult to assess new processes and project outcomes. Project management. For project implementation, project management was assigned to the Bureau of Planning in MOH, an entity that serves a coordinating function for policy, budgeting and planning in MOH, but is not well equipped to implement donor project. A more careful assessment of management arrangements during project preparation might have raised warning flags and led to introducing safeguards through appropriate 21 strengthening of the Bureau of Planning. This might have avoided some of the problems that would crop up later, especially in PHP2, in financial management and procurement under the project. In fact, all of these weaknesses would subsequently come to have bearing on the implementation, outcomes and sustainability of PHP2 and HWSP. 2.44 Quality at entry is rated moderately unsatisfactory. 2.45 Quality of supervision. While supervision monitored implementation of all project components, it focused on implementation of the key component of capacity building, where the main development impact was likely to be had; and on continued dialogue on decentralization, which carried on through PHP2 and HWS as well. Still, key constraints were not fully appreciated or acted on as they appeared: results were falling short as an ambitious menu of activities may have exceeded the clients' absorptive capacity, as the partial completion or non-completion of outputs in the efficacy section indicates; major delays continued to be experienced throughout the project period in the transfer of grant funds that were supposed to be a key ingredient in both investment planning and financial strengthening of districts. Most importantly, little consideration was given to the impact on the project of the absence of complementary legislation or regulations for decentralization, and possible adaptation of project components to that situation. Under PHP1, no major fiduciary issues arose, and while occasional financial irregularities occurred, in particular regarding procurement, these were minor and the result of unfamiliarity with Bank procedures and conflicting Bank-government procedures. They did not require declarations of mis-procurement, instead they were resolved through field-level interventions. Finally, while PHP1 should have served as a roadmap transmitting experience to the two follow-up projects - PHP1 and HWS - this opportunity was not taken. While sufficient experience may not have been gathered to be useful for PHP2, which followed close on the heels of PHP1 (only a year later), it does not seem unreasonable to assume that some experience of implementing PHP1 could have been drawn on for HWS, considering in particular the considerable uncertainties the project was facing in terms of the decentralization process. 2.46 Quality of supervision is rated unsatisfactory. 2.47 Overall Bank performance is rated unsatisfactory, given that the outcome was rated unsatisfactory. BORROWER PERFORMANCE 2.48 Government performance. While the decision to decentralize was a central political aim of the government, it left the definition of roles and responsibilities of different government levels for later - there was clarity about decentralization, but not what to decentralize or how. Such uncertainty would affect project implementation at all government levels throughout the project. Likewise, cumbersome financial transfer procedures that may not have been anticipated at the beginning, would delay project implementation and also risk project outcomes, especially as they related to the district-level learning process and the provision without disruption of health services. . 2.49 Government performance is rated unsatisfactory. 22 2.50 Implementing Agency Performance. * MOH had been given significant responsibilities under the decentralization design pursued under the project (set out in paragraph 2.6). In addressing them, it faced three principal difficulties: one, in the absence of regulation, its mandate was unclear and it was unsure of its legitimate responsibilities; two, it was not well equipped to take on a stewardship and policy-making role while continuing to manage vertical programs and providing technical support to provinces and district; and three, its resources for project management turned out to be limited. As a result, it may have been less committed to the project, which may have contributed to its modest results. The ICR points to a lack of support for the project in the ministry and that the MOH appears to have been reluctant to engage in project activities. * Provinces and districts. Implementation performance of provinces and districts provided a more mixed picture. Yogyakarta clearly responded well to the challenges offered by the project and was able to benefit from the transfer of knowledge and subsequently apply it in province and district operations. In Lampung, outcomes may have been less satisfactory, but the province (as well as participating districts), while not able to implement some of the new methods that were being introduced, was able to incorporate elements of project learning into management processes. Partial results were less a reflection of willingness to participate, and more of absorptive capacity. 2.51 Implementing agency performance is rated moderately unsatisfactory 2.52 Overall borrower performance is rated unsatisfactory. Monitoring and Evaluation 2.53 Design (see also paragraph 3.31): M&E design included arrangements for monitoring project implementation and institutional development; it did not include the grant element. Evaluation design drew on outside resources, which carries the risk of not sufficiently institutionalizing the monitoring and evaluation function. More generally, the formulation of the results framework reduced possibilities for rigorous monitoring of outcomes. 2.54 Implementation (see also Section on Monitoring Design and Implementation above). Implementation of M&E proved workable at the province and district level. At the center, problems were encountered both with the central coordinating unit, notably as a result of its limited capacity to undertake project management; and with the effectiveness of the intended longer-term arrangements (for decentralization and policy), which did not receive the necessary support in the ministry to be able to thrive. 2.55 Utilization. The M&E mechanisms that were set up for the project were used at the province and district levels to monitor quality in planning and investment proposals, and this practice has been kept up in Lampung and Yogyakarta. The arrangements built into the central ministry do not appear to have been extensively used to guide policy. 2.56 The overall quality of monitoring and evaluation is rated negligible. 23 3. Provincial Health Project 2 (PHP2) Objectives, Design, and Relevance OBJECTIVES 3.1 The Project Development Objective for Provincial Health Project 2 (PHP2) is similar to that for PHP 1, except that three new provinces - North Sumatra, West Jawa, and Banten - are targeted. 3.2 The Development Credit Agreement (DCA) stated that the Project Development Objective was "to assist the Borrower in bringing about effective health sector decentralization in Banten, North Sumatra and West Jawa and at the national level." 3.3 The Project Appraisal Document (PAD) listed two objectives: * "to bring about effective health sector decentralization in the provinces of North Sumatra, West Java and Banten." * "to help the Ministry of Health and Social Welfare (MoHSW) carry out its roles in a decentralized system." 3.4 The two statements of development objective are consistent, as they were in PHP 1. The Project Development Objective in Annex 1 of the PAD is identical to the objective in the text. The definition for "effective health sector decentralization" is the same for the two projects and includes (i) protecting health services that are essential for the poor and the public at large; (ii) initiating key health sector reforms; and (iii) putting health financing on a firm footing. The role of the MOH is defined more broadly than in PHP 1, including the provision of sector vision and leadership; analysis and policy follow-up of the health needs of the poor and risks to the public at large; advocacy of best practice and standards; and support for local initiative. 3.5 This evaluation will assess PHP2 on the basis of the objective in the PAD, including the definition of "effective health sector decentralization" and the broader role for the MOH. RELEVANCE OF THE OBJECTIVE 3.6 The relevance of the objective was high. As in PHP1, this project directly supported government and CAS objectives of effective decentralization. It supported key elements of the government's long-term strategy for the sector, "Healthy Indonesia 2010" concerning functional decentralization, and related human resource development. It addressed two of the four pillars in the World Bank's CAS for Indonesia covering the years 2005-2011, including, in particular, pillars relating to better social services delivery to the poor and to governance and implementation of the decentralization policy. 7 However, the focus on human resource development was less in this project than in PHP1 and in the subsequent Health Workforce and Services Project (reviewed after this project) 24 3.7 While the project explicitly addressed major challenges in the health sector (listed in the section on Background and Context), it was anticipated that successful decentralization would, over time, increase access to better quality services and improve healthy behaviors among the population, which in turn would reduce those challenges and accelerate improvement in outcomes. DESIGN 3.8 The project followed close on the heels of PHP1 and had a similar design. It introduced institutional reforms and strengthened financial resource mobilization in the provinces of North Sumatra, West Java, and Banten; and it continued (and expanded on the areas of) assistance to the MOH that had been started under PHP 1. It included a large grants component. As was the case for PHP1, the project was to proceed in two phases. A first phase would focus on putting into place necessary province and district level support mechanisms (JHCs, TFs, TRTs) and launch capacity building activities. The second phase would fund district health services improvement initiatives based on the capacity building that had taken place during the first phase. 3.9 Project components were the following: Component A.1. Managing Decentralization ($20.7 million; actual $8.1 million) * Project-supported Task Forces were to focus on critical institutional issues and develop recommendations to be implemented during project implementation. Topics included: (i) restructuring health services organization in the provinces and districts; (ii) strengthening human resource management policies; (iii) introducing new health information systems; (iv) upgrading the regulatory framework including the licensing and certification of health professionals, and accreditation of health care organizations; (v) strengthening public accountability mechanisms; and (vi) fostering health promotion. Component A.2. Resource Mobilization ($2.43 million; actual $0.2 million) * Support was to be provided to District Health Offices working individually or through the Task Forces to explore ways of financing district health needs. This included both making the case for adequate allocations from the district health budget, and relying on user fees whenever the population had the ability to pay for treatment costs. It would also include better targeting of public resources to the poor; transforming public hospitals into district enterprises; and testing payment systems. Component A.3. District Funding Allocations ($850.44 million; actual $761.8 million) * Project resources would be provided to districts in the form of block allocations (grants) with the aim of raising government health spending to levels that are sustainable over the medium term; a target of at least 15 percent of district budgets would be allocated to health by the end of the project. For the use of project resources, province-level Joint Health Councils would set annual block allocation 25 ceilings for districts according to pre-set criteria, including estimates of sustainable spending. Districts would then prepare four-year health improvement frameworks to support financing requirements, with a particular focus on improved coverage and quality of public health services and better curative care for the poor. Component A.4: Project Management ($5.03 million; actual $2.4 million) * Project management units would be established at province and district levels. Funds were allocated to support consultant services, supplies, and supervision. Component B.1. Institutional Development ($1.92 million; actual $0.5 million) * Establishment and support to a small Decentralization Unit within the MOH. It would provide a central entity for facilitating and coordinating the decentralization process. Component B.2. Strengthening Communicable Disease Control (US$8.57 million; actual US$3.0 million) * Assistance to transform the Directorate General for Communicable Disease Control into a specialized central agency within the Ministry of Health; this would strengthen the epidemiological surveillance system at each level and develop cross-level networking. Component B.3. Ensuring Equity in Service Provision ($3.48 million; actual $0.7 million) * The funding of National Health Grants to reorient district health systems, provide cost-effective public health programs and cofinance catastrophic care for the poor. Funds were envisaged to be directed to roughly 20 poor districts to reorient their health systems, implement cost-effective public health programs, and protect the poor from catastrophic illness by providing basic hospital services. Component B.4. Assessing Health Research Needs, Capacities and Options ($0.06 million; actual $0 million) * Support of a broad analysis of health research requirements and options within a decentralized system, including the evaluation of the capacity of the existing National Institute of Health Research and Development, all with a view to improve the outflow and use of high quality research findings. Component B.5. Strengthening Food and Drug Control ($2.01 million; actual $3.0 million) * Assistance in the transformation of a newly created food and drug agency into a professional unit, whose operations are based on scientific, risk management principles. Component B.6. Project Management ($1.79 million; actual $4.9 million) 26 * Establishment of a coordination unit at the central level for overall coordination of the project. Project funds would be used to support consultant services, office equipment and supplies, workshop, project management training, and project supervision. RELEVANCE OF DESIGN 3.10 The relevance of design is rated modest. The project had a design similar to that of PHP 1: defining the roles of the three government levels; addressing institutional barriers and capacity constraints to undertaking those roles; and it piloted new institutions through a large grant mechanism. And, while the design here too was based on concepts of good practice, it ventured into the same areas of regulatory uncertainty regarding the sharing of functions between government levels as in PHP 1. 3.11 It also suffered from similar design problems as PHP 1. The scope of the project may have been too ambitious considering the limited experience and capacity that provinces and districts had in taking on the extensive responsibilities that were envisaged under the project. Project management initially lay with the Bureau of Planning in the MOH, which now ended up having even less capacity. (Authority was later moved to a different unit in the ministry.) The results framework was similar to PHP 1, with process-oriented output indicators and broadly stated outcomes that provided little guidance for assessing outcomes or drawing up robust monitoring and evaluation designs. 3.12 Monitoring and Evaluation Design. The challenges to M&E in PHP 1 and PHP2 were the same, and the arrangements similar. Progress on institutional changes at all three government levels needed to be monitored and the results assessed; and this was also the case with increased health spending (the investment grants). The project required monitoring at three different government levels; it required a conventional focus on inputs; as well as monitoring progress and assessing results of a large number of institutional changes. Again, a relatively late introduction of output indicators, unclear links between outputs and outcomes in the results framework, and the broad definition of outcomes, would make measurement of progress and achievements difficult. Monitoring arrangements were similar to PHP 1: Bank teams and local entities would issue input-focused reports; and entities at the center and at the subnational levels would monitor and evaluate key elements of the decentralization process. Under this project, a specific decentralization unit was introduced in the MOH for this purpose, replacing the cross-ministry arrangements under PHP 1. Again, an independent research team, the same one as for PHP 1 and recruited by the MOH, was to determine measurable indicators for the results framework, undertake baseline, project midterm and final evaluations, and perform special studies. 3.13 Implementation arrangements. As in PHP 1, implementation arrangements were designed to mitigate the risks inherent in introducing new concepts into health administrations that had little experience with decentralization. For day-to-day project implementation, implementation units were set up at district and province levels. Joint Health Councils were set up at the province level to determine broad health policy for the province and award project grants. These district and provincial bodies were supported by task forces for institution building and technical review teams for support in planning and developing grant proposals. At the central level, a Central Coordinating Unit was located in the Secretary 27 General's office. The central MOH also was strengthened by policy and decentralization units that were supposed to become permanent features of the ministry structure. Project management manuals and grant operational manuals served to guide project implementation. Table 3.1. Provincial Health Project 2 (PHP2) Key Dates Process Date Process Original date Revised/actual date(s) CONCEPT REVIEW: 12/21/2000 EFFECTIVENESS: 04/24/2002 04/24/2002 APPRAISAL: 04/18/2001 RESTRUCTURING(S): APPROVAL: 06/26/2001 MID-TERM REVIEW: 10/15/2005 01/11/2006 CLOSING: 06/30/2007 06/30/2007 Source: ICR for PHP2 Table 3.2. Provincial Health Project 2 (PHP2) Project Costs by Component Appraisal Actual/latest Percentage Components estimate (US$ estimate (US$ of appraisal million) million) A.1. Managing Decentralization 20.07 8.09 40.3 A.2. Resource Mobilization 2.43 0.20 8.23 A.3. District Funding Allocation 850.44 760.79 89.46 A.4. Project Management 5.03 2.36 46.92 B.1. Institutional Development 1.92 0.53 27.6 B.2. Strengthening Communicable Disease 8.57 3.0 35 Control B.3. Assuring Equity in Service Provision 3.48 0.74 21.3 B.4. Assessing Health Research Needs, 0.06 0.00 0 Capacities and Options B.5. Strengthening Food and Drug Control 2.01 2.96 67.9 B.6. Project Management 1.79 4.95 361.6 Total Project Costs 895.81 783.62 87.4 Source: ICR forPHP2 Table 3.3. Provincial Health Project 2 (PHP2) Financing Appraisal Actual/latest Percentage of Source offunds estimate estimate (US$ million) (US$ million) appraisal Borrower 793.2 783.6 98.9 International Bank for Reconstruction 63.2 45.0 71.1 and Development International Development Association 40.0 12.0 30 (IDA) Source: ICR for PHP2 3.14 Implementation experience. Similar factors influenced implementation in PHP1 and PHP2: (i) there were differences in implementation capacity between provinces, and while it is difficult in the absence of adequate indicators or evaluation data to determine how such differences influenced project outcomes, output indicators do show variations in results between provinces; (ii) the lack of clarity in functional responsibilities thwarted some efforts, 28 notably in institution building and human resource development; (iii) much money was spent on district funding (grants) with apparently little oversight; and (iv) disbursements from the Ministry of Finance continued to be a (budget technical) problem that could not be overcome throughout the project period. In addition, (iv) fiduciary issues would become prominent in PHP2 - fraudulent practices at the central project coordinating level led to a cancellation of some Bank funds and loss of momentum in project implementation; and procurement issues, which had been relatively minor under PHP1, became more important under the second project, requiring ring-fencing, which led to a cumbersome and extensive Bank review process that also slowed down the project. 3.15 On the positive side, the province and district level support mechanisms for policy planning and investment that were introduced under the project - the JHCs, TRTs, and TFs - did facilitate the grant request preparation process, including promoting, evaluating, and advising on district investments; increased cross-district communications; and supported district planning and budgeting. There is, however, no way to assess whether outcomes were commensurate with such inputs. Monitoring and Evaluation 3.16 Monitoring of project inputs drew on the arrangements put in place for tracking procurement, disbursements, and physical deliverables. There appears to have been no monitoring of the grants component, nor were any evaluations of outputs/outcomes available. Monitoring did pick up on problems in procurement and disbursement, mentioned in the Section on Safeguards and Fiduciary Compliance. Monitoring results as compared to plans and agreed objectives became the responsibility of the independent research team that had developed the indicators. The project had envisaged baseline, midterm, and final evaluations to be undertaken by the research team, but discontinuity in its leadership resulted in incompatible evaluation results. This also may have reflected shortcomings in the design phase: the MOH did not have the technical resources to manage the research team, nor was it apparent that the ministry had been sufficiently engaged in developing indicators - a weakness that was also present in PHPl. 3.17 Safeguard compliance. The only safeguard policy concerned indigenous people. The policy was triggered to address district health proposals in the province of Banten, which has a significant indigenous population. There, designs were to take into consideration the social and cultural environment of the indigenous population. No issues arose during project implementation, as (i) local authorities were able to draw on research that had been undertaken by a local university and funded by a Bank project (Third Community Health and Nutrition, Loan 3550-Ind); and (ii) the indigenous population was already seeking out public health services for care. 3.18 Fiduciary compliance. Fiduciary issues arose during project implementation concerning financial practices in the Central Project Coordinating Unit of the project. (This unit also was the central coordinator for PHP 1.) The Bank took prompt action to remedy the situation, introducing a series of preventive measures that would apply to all three projects. These prevented a recurrence of instances of fraud in the projects. However, these actions came too late to avoid substantial damage to PHP2. They resulted in protracted procurement 29 processes to ensure that Bank requirements were being met; they led to reimbursement by the government for ineligible expenditures to the amount of 8.7 million SDRs; and momentum was lost in project implementation, leading to cancellation of $32 million of the loan amount at the request of the government; effectively truncating the project. The MOH dismantled the Central Coordinating Unit and reestablished it in another section of the ministry to ensure continued coordination of the three projects. 3.19 Procurement. Procurement, particularly in the second project, was a further weak element in project implementation. Low capacity and lack of understanding of World Bank procurement processes, mainly by district implementing units, was widespread and raised the question of whether procurement under the project should have taken place at the district level at all. However, given the emphasis on decentralization, it was appropriate to consider procurement as one element of decentralized management. That said, procurement responsibilities at central, province, and district levels should have been better defined, and Bank support designed accordingly. It appears that the focus was on the fiduciary obligation related to the use of Bank funds for procurement. More caution might have been exercised with, at least initially, a more selective approach being applied as to what could and could not be procured locally, an analysis which could have formed a stronger basis for sharing functions between government levels, as well as for defining capacity building needs. In the event, the Bank did recognize capacity constraints and responded with low thresholds for prior review. This of course resulted in a large volume of prior review contracts, involving all three projects. And continued irregularities throughout the project cycle made it impossible to reduce the Bank's review workload. Achievement of the Objectives Objective 1. To Bring about Effective Health Sector Decentralization in the Provinces of North Sumatra, West Java, and Banten 3.20 Achievement of the objective is rated modest. 3.21 As was the case in PHP1, in support of Objective 1 - "effective decentralization" - the following outcomes were specified: (i) protecting health services that are essential for the poor and the public at large, (ii) initiating key health sector reforms, and (iii) putting health financing on a firm footing. The rating for achievement of Objective 1 is based on the extent to which these outcomes were reached, set out in the bullet points below and discussed in paragraphs (i) Protecting health services that are essential for the poor and the public at large * Implementation of effective, locally relevant curative service delivery, and public health service provision and financing arrangements. Partly implemented. Three rounds of annual district grants for strengthening health services were prepared following evidence-based planning, compared to a target of four rounds. 30 * Contact rate of poor people in the health center system in North Sumatra, West Java, and Banten reaches at least 2 times a year by January 1, 2006. Partly implemented. Health service utilization was used instead, because reliable contact rate data were not available. Specifically, data on utilization of public facilities by the poor was used. Between 2002 and 2006 utilization by the poorest quintile increased from 24.8 percent to 29.2 percent in North Sumatra; decreased from 40.9 percent to 38.2 percent in West Java; and increased from 49.0 percent to 53.5 percent in Banten. For the second quintile, utilization increased from 21.4 percent 28.3 percent in North Sumatra; decreased from 40.9 percent to 38.9 percent in West Java; and increased from 44.3 percent to 49.2 percent in Banten. (ii) Initiating key sector reforms * Joint Health Councils and Technical Review Teams set up and operating at the province level; and Task Forces set up and operating at district levels to strengthen policy and investment planning and implementation. Implemented * Implementation of new personnel policies related to recruitment, redeployment, retraining, early retirement, and career development for North Sumatra and West Java by January 1, 2004, and for Banten by January 1, 2005. Not implemented. North Sumatra and West Java provinces conducted review of existing personnel policies in 2004. Nine districts in West Java conducted similar review, but only one district introduced new policy according to local needs. * Reduction of the number of non-health staff at the provincial and district health levels by at least 15 percent compared to staff numbers as of January 1, 2001 in North Sumatra and West Java by December 31, 2005, and in Banten by December 31, 2006. Not implemented - not in line with national policy * Implementation of decentralized, province specific health information systems for North Sumatra, West Java, and Banten by January 1, 2004, and for Banten by January 1,2005. Not implemented. North Sumatra conducted review of existing HIS in 2004 and develop a blueprint for an integrated health information system in 2006 but it is not yet implemented. West Java only prepared instruments for collecting baseline data. * Implementation of new licensing and accreditation system (for health staffing and health facilities) for North Sumatra and West Java by January 1, 2003, and for Banten by January 1, 2004. Partly implemented. North Sumatra and West Java designed an independent Quality Council to be responsible for quality assessment of providers in 2006. 31 * Implementation of at least two public accountability mechanisms for North Sumatra and West Java by January 1, 2003, and for Banten by January 1, 2004. Partly implemented. By 2006, a healthy district forum was implemented in 16 districts in North Sumatra. By 2006, 15 districts in West Java preferred to continue with the implementation of a complaint resolution mechanism. * At least five health promotion proposals from NGOs are selected by the Health Promotion Boards and financed by the project in North Sumatra and West Java, Partly implemented. Only North Sumatra was successful in establishing a health promotion board, and it was funding more than five NGO proposals per year every year in North Sumatra, beginning in 2003, and in Banten every year beginning in 2004. (iii) Putting health financing on a firm footing * At least 80 percent of district hospitals and 40 percent of health centers in West Java, at least 50 percent of district hospitals and 30 percent of health centers in North Sumatra, and at least 50 percent of district hospitals and 20 percent of health centers in Banten, become semi-autonomous (partially self-financing) units. Partly implemented. By project closing, 8 percent of district hospitals and 4 percent of health centers in North Sumatra, 56 percent of district hospitals and all health centers in one district in West Java, and 20 percent of district hospitals and no health centers in Banten had become semi-autonomous. * At least 30 percent of the population in North Sumatra and West Java, and at least 20 percent of the population in Banten are covered by health insurance mechanisms or managed care by December 31, 2006. Implemented. Populations covered by various insurance schemes in 2006: North Sumatra 29.3 percent, West Java 32.7 percent, and Banten 36.0 percent. * Health spending increases at the district level between 2001 and 2006, reaching 15 percent of total local government spending by 2006. Partly implemented. Only 3 districts in West Java achieved the minimum 15 percent target in 2006. In other districts, recording and reporting was too unpredictable to draw conclusions. 3.22 As was the case in PHI, indicators were not sufficiently well articulated to allow determination of the extent to which each outcome was achieved and thus the extent to which Objective I was achieved. At best, district grants (supplementing district spending) should have had an important effect on objectives (i) and (iii), at least during project implementation - but there is no assessment available on the impact the grants really had. Sub-national support mechanisms had not been included as an outcome indicator, contrary to the case in PHI. Nonetheless, this capacity building objective was part of the project, and it showed similar mixed results: where capacity already existed, notably in West Java, the mechanisms 32 were welcome and became part of the institutional setup; in other provinces, less so. Again, as in PHP 1, indicators were mostly either partly implemented, or not implemented. At least to some extent, this may reflect a combination of low absorptive capacity among clients and an ambitious project. 3.23 Protecting health services. As was the case in PHP 1, of the two indicators grouped under outcome (i), the first one, on district grants, should stand on its merits, but the absence of information on actual spending and outcomes, reduces its value as an efficacy measure. The indicator on utilization rates should not be interpreted as unambiguously being due to decentralization only. In particular, the introduction of public assistance for health - unrelated to the project - certainly also has contributed to higher numbers. 3.24 Initiating key sector reforms. The sub-national support mechanisms provided an important input into strengthening decentralization, which may well have been the main, albeit significant, contribution toward sector reform. No other initiatives were fully implemented in all provinces, although new quality control mechanisms were put into place in North Sumatra and West Java. 3.25 Health financing. District grant funding may have played a role in stabilizing and increasing health financing simply by the significant amounts that were transferred - some $760 million, but there is no way to tell. At the same time, district health spending rose to 15 percent of the budget in West Java; while data is not available from other provinces, reflecting one result of decentralization - weak reporting of health spending at district levels. Objective 2. To Help the Ministry of Health and Social Welfare Carry Out its Role in a Decentralized System 3.26 Achievement of the objective is rated modest. * A Decentralization Unit is set up and functioning at the central level by December 31, 2001. Implemented * Establishment of Geographic Information System and Early Warning Outbreak Response System for communicable diseases in Banten, North Sumatra, and West Java by June 30, 2004. Partly implemented. Achievement of a Memorandum of Understanding between the center and Head of Health Office in each district * A public health laboratory for environmental health and communicable disease control are established in Jakarta and Medan by December 31, 2003 Implemented * An enhanced inspection system related to food and drug control is in place by December 31, 2003. 33 Implemented * A food and drug complaint resolution mechanism is established within regional food and drug control offices by December 31, 2003. Implemented 3.27 The project complemented initiatives that had been undertaken under PHP1 to enhance MOH's role in analysis and advocacy. The intra-ministerial decentralization arrangement giving most ministry directorates a role had not been successful, and it was replaced by a specific decentralization unit to coordinate the ministry's support for decentralization. Its impact appears to have been modest: it turned out to have limited experience and leverage for leading the reform process in the ministry. The other initiatives, while important technical inputs into the ministry's functioning, were not central to the decentralization process. Efficiency 3.28 Project efficiency is rated negligible. While no net present value or rate of economic return was calculated for the project in the absence of relevant data, benefit-cost ratios were estimated at project preparation. The main benefit was to be an improvement in health status (number of life-years saved) that would result from improvements in allocative efficiency in spending and improved access to better quality health services. A minimum-benefit scenario needed to justify project investments was estimated as a 2.5 percent increase in coverage of services. No post-project benefit-cost analysis was undertaken to determine if these numbers were achieved. Some other measures that might have contributed to greater efficiency - in health financing, organizational and administrative changes, or personnel policies - were only partially achieved or not achieved at all. There is no evidence of the impact of the 90 percent of project funding ($761 million) that was transferred by the government to districts through grants to support their annual plans. Project efficiency was further reduced by perennial delays in releasing funds from the Ministry of Finance, which disrupted implementation of district plans. Finally, fraudulent practices relating to procurement and financial management led to cancellation of some $32 million of the loan amount and resulted in a truncated project. Ratings OUTCOME 3.29 The outcome is rated unsatisfactory. 3.30 The relevance of the project development objective is rated high, as it directly addressed a key policy variable - decentralization - that featured prominently in the country's development strategy and in the Bank's country assistance strategy. Relevance of design is rated modest. While it had the merit of trying to build an operational framework for health sector decentralization, including drawing on models of good practice, the design anticipated decisions that either would not be forthcoming, or would be only partially made. 34 Moreover, the results framework was insufficient to allow a rigorous assessment of project outcomes. Efficacy is rated modest for both objectives, since few of the targets pertaining to decentralization were met. Efficiency is rated negligible, reflecting the apparently modest effects on institution building and the absence of results for financial transfers.8 Finally, fraudulent practices resulted in a truncated project. RISK TO DEVELOPMENT OUTCOME 3.31 At appraisal, the risk to the development outcome was rated modest, and in the ICR it was rated substantial. Risks were both political and institutional. While the necessary legislation had been put in place, the actual form decentralization would take in terms of the roles of the three government levels and the sharing of functions, remained undefined. The project, as well as PHP1, was a way of helping the Government move forward on these issues. Here, the risk was that the project approaches and the knowledge transmitted through the project would not be sustainable, if and when more detailed legislation took shape. As has been noted throughout the document, decisions in key areas of service organization and resource management still have not been taken. And some of the initiatives that were introduced under the project have not been sustainable - for instance the policy planning processes, the autonomy of health facilities, and so on, have taken other forms. Districts have gone ahead with elements of decentralization, especially in operating the service network and in local planning and budgeting. The role of the province has become somewhat different from that envisaged under the project - it essentially function as an extension of the central Ministry, but also in some instances plays a coordinating role for the districts. The central Ministry has maintained many of its previous prerogatives, especially in human and financial resource management; and it has been slow to follow up on major elements of support provided by the Bank, notably the policy and decentralization functions. On the above basis, the risk to the development outcome as defined in the project is rated significant. BANK PERFORMANCE 3.32 Quality at Entry. The quality at entry was similar to that in PHPl. It was strong in the following areas. The project was strategically relevant - it focused on decentralization - and the approach was to address major challenges to implementing decentralization. It was underpinned by analytical work. Technical design was sound - it focused on building up institutions at all three government levels in anticipation of decisions on roles and responsibilities, and it included important national-level initiatives to strengthen food and drug safety, and communicable disease management. It included a financial and economic analysis; it attempted to introduce poverty criteria in district funding allocations, and health promotion to influence healthy behaviors among the target population. Institutional innovations were introduced up front to allow for quick project start-up and ample time for adjusting new processes as necessary during implementation. The latter allowed for a project design that otherwise might have been overly ambitious. A rough estimate indicates a 10 percent increase in coverage of public health programs and skilled birth attendance over the project period at a cost of US$760 million (the value of the district funding allocation) 35 3.33 Quality at entry was weak in its assessment of project risk, the absorptive capacity of the client, in monitoring and evaluation, and in administrative arrangements for implementation. Project risk. While project risks were assessed as substantial, the risk assessment did not address the implications that uncertainties about roles and responsibilities of the three government levels in decentralization might have for the success of the project. Absorptive capacity. The same uncertainties would also affect absorptive capacity: all three government levels were already struggling with the often unclear responsibilities imposed by the decentralization law, on top of which the project introduced innovations that still had little formal backing in law or regulation. Monitoring and evaluation. The formulation of the results framework would make it difficult to construct a satisfactory monitoring and evaluation system, and this would make it difficult to assess new processes and project outcomes. Project management. For project implementation, project management again was assigned to the Bureau of Planning in MOH. This may have been a significant contributing factor to subsequent implementation difficulties, as the Bureau now was saddled with managing two projects. A more careful assessment of management arrangements during project preparation might have raised warning flags and led to introducing safeguards through appropriate strengthening of the Bureau of Planning. This might have avoided some of the problems that would crop up later, especially in PHP2, in financial management and procurement under the project. 3.34 Quality at entry is rated moderately unsatisfactory. 3.35 Quality of Supervision. The quality of supervision of PHP2 was similar to PHP 1. While supervision monitored implementation of all project components, it focused on implementation of the key component of capacity building, where the main development impact was likely to be had. An important element of the supervision process was also continued dialogue on decentralization, which carried on through both PHP1 and PHP2. Capacity building appears to have had some success: field-level observations indicate that North Sumatra and in particular West Java provinces, as well as participating districts, have been able to build on the project's planning and investment structures for their current policy planning and budgeting. Despite this favorable post-project development, less attention was paid to results during the project life. Key constraints were not fully appreciated or acted upon as they appeared: results were falling short as an ambitious menu of activities may have exceeded the clients' absorptive capacity, as the partial completion and non-completion of outputs in the efficacy section indicates; major delays continuously were being experienced in the transfer of grant funds that were a key ingredient in both investment planning and financial strengthening of districts. Most importantly, little consideration was given to the impact on the project of the absence of complementary legislation for decentralization, and possible adaptation of project components to the situation. While the Bank initially acted decisively when allegations of corruption appeared, targeting the coordinating team in MOH, quickly developing an action plan to remedy the situation, it was not able subsequently to maintain necessary momentum among government officials to maintain the project, ultimately resulting in the cancellation of part of the credit in addition to funds that had to be reimbursed. Finally, as was the case with PHP 1, PHP2 was also not used as a roadmap to transmit experience to the next decentralization project coming up - HWS. 3.36 Quality of supervision is rated unsatisfactory. 36 3.37 Overall Bank performance is rated unsatisfactory. BORROWER PERFORMANCE 3.38 Government Performance. While the decision to decentralize was a central political aim of the government, it left the definition of roles and responsibilities of different government levels for later - there was clarity about the desirability of decentralization, but not on what to decentralize or how. Therefore, it is not clear to what extent the roles identified in the project for different government levels had full government support, or even the support of the main implementing agency, MOH. Implementation responsibility fell on MOH, which would turn out to have limited capacity for effective overall implementation. Little attention appears to have been paid to the mechanisms for transfer of fiscal grant resources, a crucial feature of the project. The cumbersome procedures of the Ministry of Finance and the significant delays these caused in transferring funds were features that were well known and sure to disrupt progress on the project. While these features applied to both projects, allegations of corruption added to the PHP2. Here, the response was initially prompt, but follow-up actions slow. Moreover, there was subsequently little enthusiasm for improving project performance, according to the ICR. Instead, the government settled for a set of limited results - a series of "partially implemented" actions, and let the project run out its remaining time 3.39 Government performance is rated unsatisfactory. IMPLEMENTING AGENCY PERFORMANCE 3.40 Implementing agency performance includes the MOH and province and district project entities. * Ministry of Health. The PHP2 project faced the same challenges in MOH that had affected implementation of the first project - an unclear mandate, limited capacity for its new role, and difficult project management. The introduction of a decentralization unit to support the decentralization process proved insufficient to provide necessary support to provinces and districts. * Provincial and District Project Entities. Here, the picture was more mixed. Some provinces and districts, such as West Java, made significant efforts to implement the project in accordance with their understanding of project objectives. The results are still noticeable to this day, where current teams continue to benefit from the capacity building under the project. There is no information on commitment or achievement in the other provinces. 3.41 Implementing agency performance is rated unsatisfactory. 3.42 Overall borrower performance is rated unsatisfactory. The ratings reflect the ratings for government and implementing agency performance. 37 MONITORING AND EVALUATION 3.43 Design (See also paragraph 3.32). While the results framework limited the scope for rigorous, outcome-oriented M&E, arrangements nevertheless were in place for tracking inputs, as well as for monitoring progress on institution building, notably investment planning. Initially, design did not include measurable indicators; these were added during implementation by consultants. The PAD included only composite indicators expressed in general form. What indicators there were to monitor related to project outputs. 3.44 Implementation. Implementation proved workable at the province and district levels where project entities were able to monitor and evaluate the district planning process, albeit with mixed results for quality. Project indicators - baselines, milestones, targets - were developed by consultants only when the project was well into implementation. Moreover, the MOH appears not to have had a qualified counterpart to engage in technical discussions and provide direction to the research unit. 3.45 Use of Data. While the quality of district planning was documented, indicators did not capture well whether or how institutional development actions improved efficiency and capacity; and there was little use of monitoring and evaluation data for decision-making, beyond measuring progress on the implementation of institutions. 3.46 Monitoring and evaluation is rated negligible. 4. Health Workforce and Services Project (Provincial Health Project 3) - HWS Objectives, Design, and Relevance OBJECTIVE 4.1 According to the Development Credit Agreement (DCA), the project objective was to "assist the Borrower in achieving the effective delivery of health services in Indonesia in a decentralized setting by strengthening (i) the financing and delivery of health services in the Borrower's provinces of Jambi, East Kalimantan, West Kalimantan and West Sumatra, so as to enhance the quality of care and health outcomes at the District level; and (ii) health workforce policy, management and development at the national and sub-national levels so as to improve allocational efficiencies and equity in the distribution and use of health resources." 4.2 The Loan Agreement does not state the project's development objective; instead, it refers back to the DCA. 4.3 The Project Appraisal Document (PAD) includes a general development objective, as well as specific ones. The general development objective was "to support health sector decentralization in four provinces for sustainable financing and client-centered delivery of health services." 38 4.4 The specific development objectives were to: "(i) improve financing and delivery of essential health services in the provinces of Jambi, East Kalimantan, West Kalimantan, and West Sumatra to enhance access to care, quality of care and health outcomes at the district level; and (ii) strengthen health work force policy, management and development in a decentralized context in order to improve allocational efficiencies and equity in the distribution and use of health resources in the districts." 4.5 A "corollary" development objective was "to empower the Ministry of Health, the Ministry of National Education, and the Indonesian Medical Association (IMA), the three key stakeholders in the sector, through: (i) assistance to redefine their roles and responsibilities vis-a-vis health work force policy, planning and management; and (ii) building their institutional capacity for effective stewardship in fulfilling the functions of policy making, legislation, regulation, quality assurance/control and technical assistance to provinces and districts." 4.6 The objectives in the Development Credit Agreement and the Project Appraisal Document are materially consistent; both emphasize activities and outputs that support the achievement of the project objective "to assist the Borrower in achieving effective delivery of health services [...] in a decentralized setting." Both documents then go on to describe "effective delivery" as being "[enhanced] quality of care and health outcomes at the district level" and consisting of "[improved] efficiencies and equity in the distribution and use of health resources." 4.7 This review will assess the achievement of the PDO as stated in the DCA. RELEVANCE OF THE OBJECTIVE 4.8 The relevance of the objective is rated high. As was the case with the two Provincial Health Projects, PHP1 and PHP2 (P049545, P049539) that preceded HWS, the project directly supported government and CAS objectives of promoting effective decentralization, responding to a key feature of the government's long-term strategy for the sector, "Healthy Indonesia 2010"; and in the Bank's Country Assistance Strategy for Indonesia covering the years 2005-2011. It also addressed a second key feature - human resource development, focusing on both the quality and distribution of the health workforce. DESIGN 4.9 The project's decentralization design built on PHP1 and PHP2, both of which were underway in other provinces. Like those projects, HWS aimed at introducing institutional reforms and strengthening financial resource mobilization in participating provinces; and at continued strengthening of stewardship capabilities at the central level. At the same time, it had a stronger focus on health workforce development and medical education. 4.10 The project components were as following: Component A: District Health Offices and Health Facilities Estimated cost at appraisal: $374.67 million; Loan/Credit Amount $74.5 million; actual cost $39.06 million. 39 Component Al. Improved access to and quality of health services. The project would help build better and more equitable health services at the district level, drawing on health development master plans and annual health plans developed by the district and financed through block grants. Proposals were expected to address (i) access and quality of services; (ii) equity and sustainability; and (iii) means of strengthening decentralized management. Component A2. Project management. Project management would be the responsibility of District Implementation Units under the supervision of the district government. Component B: Provincial health Offices and Health Facilities Estimated cost at appraisal $28.96 million; Loan/Credit Amount $9.07 million; actual cost $5.53 million. Component B.1. Health workforce development. The project would support the Provincial Health offices in providing technical and administrative support to the districts to strengthen health workforce management and training capacities. This would include assisting in developing provincial capacity to support districts in: (i) adapting the minimum service standards to local conditions and means; (ii) determining the numbers, skills, and specializations of existing and future health work forces needed at provincial and district levels; and (iii) implementing measures to ensure appropriate numbers of adequately trained staff in the health facilities. In addition, the project would support province-level studies, pilot tests, and operational research concerning incentives and motivation, performance appraisals, and other initiatives for improving workforce performance. Component B.2. Health system coordination, planning, and management. The project would build capacity for the province to support districts in other areas. These included: (a) developing guidelines for health information and education campaigns; (b) monitoring the epidemiological situation; (c) cross-sector cooperation and health development planning; and (d) health systems financing. Component B.3. Project management. A provincial coordination and implementation unit would be established at the provincial level, with responsibilities in the areas of planning, procurement, financial management, and monitoring and evaluation. Component C: Central level Estimated cost at appraisal $26.74 million; Loan/Credit Amount $22.03 million; actual cost $10.34 million. Component C.1. Effective health system stewardship. The project would continue to develop the MOH's analytical, advisory and advocacy capacity in accordance with its stewardship role in the decentralizing environment. It would also strengthen the ministry's capacity to: (i) carry out human resource development responsibilities within the context of decentralized workforce management; (ii) improve the quality 40 of pre-service education programs; (iii) ensure quality in in-service training provided at district level; and (iv) promote self-regulation in the nursing and midwifery professions. It would strengthen health workforce policy planning by strengthening strategic planning in the MOH. It would introduce a pilot program promoting competency-based training of specialists for deployment in district hospitals. And it would support the Indonesian Medical Association in developing its ability to self- regulate and empower the medical profession. Component C.2. Enhancing the quality of medical education. To better adapt to the changing health environment and consumer expectations, the project would support the Ministry of National Education to (i) increase its institutional capacity to organize and manage medical education; (ii) improve the quality of formal medical education by introducing competency-based education; and iii) enhance the learning and teaching environment for both undergraduate and post-graduate medical education and training. RELEVANCE OF DESIGN 4.11 The relevance of design is rated modest. While the project's decentralization components were as relevant to achieving the objectives of effective decentralization as were those in PHP1 and PHP2, the project was also introduced into an environment where there still was uncertainty surrounding the form decentralization might take, beyond the general guidelines that had been issued at the Big Bang. This situation, which already was present and influenced outcomes for PHP1 and PHP2, should have been even more strongly recognized when this project was designed, as little headway had been made on the sharing of functions since the two preceding projects had been launched in 2000 and 2001. Although the risk analysis draws attention to the absence of agreement on the roles and responsibilities on workforce policy and ranks that risk as "substantial," it treats it as an institutional obstacle that can be overcome using project instruments, rather than a significant political risk. 4.12 Project design was overly ambitious. While PHP1 and PHP2 had set an ambitious agenda, both focused on decentralization. To this already complex theme, HWS added human resource policy and management, which had been a challenge prior to decentralization and would remain so with decentralization. 4.13 The results framework faced the same challenges as PHP1 and PHP2. The formulation of the project development objective, especially the specific objectives, was awkward. Outcome indicators were of a general nature and not easily linked to outputs. Much of the framework consists of a listing of project components and sub-components, with related outputs and "output indicators," mainly institutional ones. They are altogether 49 in number, many of them vague and open to interpretation. In addition, a set of four health- related outcome indicators were defined to determine the health impact of the project. They raise the question of how to distinguish project impact from other influences on health outcomes. After the midterm review of the project, a set of more operational indicators were introduced which form the basis for the assessment of the efficacy of outcomes. 41 MONITORING AND EVALUATION DESIGN 4.14 While the PAD is not explicit about M&E, it indicates a traditional focus on project- based monitoring and evaluation mechanisms. The project results framework may have been too general to offer much scope for results-based monitoring or evaluation; with the exception of health outcome indicators for which baseline data had been identified and targets set, with data to be drawn from official mortality data and Demographic Health Survey data. Beyond that, the design did not clarify to what extent, or how, it would produce methodologically sound assessments of outputs, given the objectives. Implementation 4.15 Implementation arrangements. The project management structure was similar to PHP1 and PHP2, with some differences arising because of the emphasis on health workforce development. District Health Councils comprising health officials and civil society representatives strengthened governance and provided overall guidance at the district level. The Task Forces that had played an important role in planning and budgeting had been replaced by District Implementation Units that coordinated the development of annual plans within the framework of a five-year master plan. Grant proposals would be selected from the plans by the District Health Council. At the province level, the Joint Health Councils and the Technical Review Teams were maintained. A Provincial Coordination Implementation Unit served coordinated the Bank project. At the center, an Inter-agency Coordinating Committee had been set up, reflecting the fact that two ministries - the MOH and the Ministry of National Education, as well as the Indonesian Medical Association - were participating in the project. A Technical Review Committee had been introduced into the Secretary General's Office in the MOH to advise on technical matters. Project coordinating units had been set up in the MOH and in the Ministry of National Education. Table 4.1. Health Workforce and Services Project (Provincial Health Project 3) Key Dates Process Date Process Original date Revised/actual date(s) CONCEPT REVIEW: 10/02/2001 EFFECTIVENESS: 12/30/2003 APPRAISAL: 04/03/2003 RESTRUCTURING(S): APPROVAL: 06/12/2003 MID-TERM REVIEW: 01/15/2007 01/15/2007 CLOSING: 12/31/2008 12/31/2008 Source: ICR for HWS 42 Table 4.2. Health Workforce and Services Project (Provincial Health Project 3) Planned vs. Actual Disbursements Appraisal Actual/latest Percentage of estimate (US$) estimate (US$) appraisal A. District health offices and facilities 309.25 39.06 12.63 B. Province health offices and facilities B.1. Health workforce development 19.05 2.25 11.81 B.2. Health system coordination, 3.38 1.33 39.34 planning and management B.3. Project management 1.74 1.95 112.07 C. Central C.1. Health systems stewardship 9.66 4.33 44.82 C.2. Enhanced quality of medical 12.61 6.01 47.66 education Total Project Costs 429.37 57.49 13.39 Source: ICR for HWS Table 4.3. Health Workforce and Services Project (Provincial Health Project 3) Financing Appraisal Actual/latest Percentage of Source offunds estimate estimate (US$ million) (US$ million) appraisal Borrower 354.87 2.71 0.01 International Bank for Reconstruction 74.50 54.78 73.5 and Development Source: ICR for HWS 4.16 Factors affecting implementation. As was the case in PHP1 and PHP2, project implementation was affected by the uncertainties surrounding decentralization, The government passed supplementary decentralization legislation in 2004, but that did not bring additional clarity to the sharing of responsibilities. The project was affected by frequent turnover of staff and repeated delays in the disbursement of project funds, essentially block grants, to the provinces and districts. The grants were an important element in building up their planning and investment capacity. During the first three years of the project, only some 18 months were devoted to implementation, and by Closing Date, only some 13 percent of project funds had been disbursed, including 47 percent of IBRD/IDA funds. Delayed disbursements were cut short when the Government decided not to extend the project beyond the original end-December Closing Date. Consequently, some $30.8 million of the IBRD Loan and $18.1 million of the IDA Credit were cancelled, as well as $352 million of Borrower funds, mainly grants that were undisbursed at the end of the project. (See Tables 4.2 and 4.3). 4.17 Implementation of monitoring and evaluation. The implementation of monitoring and evaluation was focused on the delivery of inputs and outputs; beyond the health outcome indicators of the Millennium Development Goals, where the project effect is likely to be difficult to discern from other influencing factors, there is no documentation available suggesting that attention had been devoted to outcome monitoring. A midterm review of the 43 project drew attention to the issue, and the Bank recruited an international consultant to develop a monitoring and evaluation strategy. By that time, the project was already into its fourth year. Based on recommendations of the consultant, performance indicators were revised and information on them was collected for the final project evaluation (Central Project Coordinating Unit, 2008). The assessment of efficacy below makes use of them. 4.18 Safeguards compliance. The following safeguard policies were relevant: environmental, pest management, and indigenous peoples. Provisions were made in the project to ensure compliance with applicable Bank safeguard policies. Potential environmental and health impacts pertained to construction and rehabilitation of district level facilities, waste management, and pesticide use in vector control programs. None of these are mentioned as issues in the ICR; this was also confirmed by former project staff and interviews in the field. 4.19 Fiduciary compliance. Following the experience with fraudulent practices under PHP2, the Bank drew up an action plan to strengthen financial control of all three projects. The plan included (i) moving financial and procurement control from the Bureau of Planning to relevant technical units in the MOH; (ii) recruiting a firm to conduct internal audits of the projects and strengthen internal control in the ministry; and (iii) appointing new external auditors. Still, procurement continued to be a problem. Post reviews in 2006 and 2007 found signs of collusion and other irregularities, especially at the district level. This was in part attributed to unfamiliar Bank procurement procedures and differences between Bank and government procedures. Subsequent Bank support in applying procurement procedures through workshops and training appear to have bome fruit, and the ex post review for 2008 found that signs of collusion had decreased sharply. On the financial management side, progress appears to have been less satisfactory. The Bank's Financial Management Review, conducted in December 2008, concluded that in general "progress has been less than satisfactory - follow up actions to audit findings were not always taken, and outstanding receivables were excessive, in part a reflection of the delays in cash flows from the Ministry of Finance." Achievement of Objectives Objective 1. Improve Financing and Delivery of Essential Health Services in the Provinces of Jambi, East Kalimantan, West Kalimantan and West Sumatra to Enhance Access to Care, Quality of Care and Health Outcomes in the District 4.20 The achievement of the objective is rated modest. 4.21 Outputs: * Preparation and use of a Health Development Master Plan for all capital investment in human, financial, and physical resources in all participating provinces. Partly implemented. Development of plans initially was difficult without significant technical assistance, but skills gradually improved throughout the project period. 44 * Design and implementation of new provincial and district-level institutional mechanisms (DHCs, JHCs, TRTs) for improvement in policy making, system planning, and management Implemented * Provision of (five rounds of) block grants following applications to the MOH and based on district master plans and sector priorities. Partly implemented. To a large extent funds were used for routine activities, rather than strengthening of district health systems. * The share of district health spending to reach 15 percent of total local government spending, except in East Kalimantan where it was to reach 8 percent. Not implemented - only Jambi showed an increase in health spending as a share of the budget, from 6 percent to 8 percent. 4.22 Outcomes: * Under-5 mortality rate to decrease to 34 per 1,000 live births in all participating provinces. Not implemented in any province. The best performance was in East Kalimantan, where under-5 mortality decreased from 50 to 38 per 1,000. Jambi also saw a decrease in mortality, from 51 to 47 per 1,000. In West Kalimantan it remained flat at 59 per 1,000 and in West Sumatra it increased from 59 to 62 per 1,000 * The proportion of births attended by skilled health personnel to increase to 90 percent in all provinces. Not implemented in any province. Birth attendance declined in three provinces and remained flat in West Sumatra. * The proportion of pregnant women receiving at least four ante natal visits to increase to 90 percent in all provinces. Not implemented. The proportion of women receiving at least four ante-natal visits increased in all four provinces. It started from a fairly high baseline of between 66 and 80 percent, reaching around 80 percent in most cases and 85 percent in West Sumatra (from a baseline of 80 percent). * The segment of the population belonging to the lowest income quintile to have increased their service utilization by 50 percent. Not implemented. The increase in service utilization ranged between 20 and 40 percent, a 40 percent increase registered in West Kalimantan. 45 4.23 In summary, the project achieved none of its health-related outcome targets envisaged under Objective 1. In some instances there was progress, but it is difficult to distinguish possible project-related effects from other developments in the health sector over the project period, notably the continued expansion of the service network as a result of the Special Budget Allocation (Dana Allokasi Khusus), the introduction of the health poverty benefit, the rapid increase in graduates from new education and training institutions, and government incentives to draw new graduates into rural health systems. Institution building was not sustainable in the originally envisaged form, but provinces and districts may have benefited from capacity building and the preparation of master plans, although there are no indicators to establish that. Likewise, there is no analysis of the impact of the district funding allocations. According to the ICR (page 12), the block funding was mainly used to finance routine activities and management, less so for purposes that aligned with the project development objective. Objective 2. Strengthen Health Workforce Policy, Management, and Development in a Decentralized Context in Order to Improve Allocational Efficiencies and Equity in the Distribution and Use of Health Resources in the Districts 4.24 Achievement of development objective 2 is rated modest. 4.25 Outputs (no outcomes were specified): * The Indonesian Medical Association (IMA) to have established an institutional structure and instruments for a national medical examination and certification, for formal, graduate, and continuing medical education. Implemented. Guidelines for IMA institutional structure completed, National Medical Examination System and Certification available. * The portion of medical students in Mulawarman and Andalas Universities surveyed as satisfied with the curriculum and education provided is at least 80 percent. Partially implemented - some 70 percent of students satisfied. * Development and implementation of new standards for licensing, certification, and registration of health professionals and accreditation of health facilities in participating provinces. Partly implemented - Competency-based standards for doctors, dentists, and midwives were established at the national level during HWS and each of the four provinces made considerable progress in implementing new standards for licensing, certification, and registration of health professionals in ways reflecting the new emphasis on demonstrated competence. * Adoption of policies and financial and non-financial incentives by provinces and districts to increase productivity, performance, and motivation of health professionals. 46 Partially implemented - within the constraints set by civil service regulation and available resources at the district level, districts could adopt intended policies. Its longer-term usefulness in light of absence of significant changes in staffing policy makes the usefulness of this indicator questionable. * Redefinition of MOH roles and responsibilities relative to health workforce policy, planning, and management. Not implemented. So far, the roles and responsibilities of the central ministry have remained unchanged from pre-decentralization in this area. * Strengthening of MOH's institutional capacity through structural changes in organigram, and increased allocation of human and financial resources for effective stewardship in fulfilling the functions of policy making, legislation, regulation, quality assurance/control, and technical assistance to provinces and districts. Not implemented - an insufficiently defined element of the project that held little interest for the MOH. 4.26 Workforce initiatives at the central level were not successful. The building up of the Indonesian Medical Association as a self-regulating entity through the project turned out to be difficult and was only partly achieved. While an information system for health sector personnel was developed, it was insufficient as a basis for policy planning and management, and with insufficient reporting, it has remained of limited use. The only unambiguous success under this objective has been the development of a competency-based curriculum for medical education. In an independent evaluation (PT Bahana Mitra Buana and others 2008) of first-year medical students at two medical schools (in East Kalimantan and West Sumatra), over 80 percent of first-year students expressed satisfaction. 4.27 Competency-based standards for doctors, dentists, and midwives were established at the central level during project implementation, and these were being introduced gradually in the four project provinces (as well as in other provinces). While the project had encouraged human resource planning at the province and district levels, and plans had been developed for use of staff, sub-national levels had little scope for independent action in personnel policy. This would continue to be the case after the project had closed, rendering this project initiative only marginally meaningful. Efficiency 4.28 While no net present value or rate of economic return was calculated for the project in the absence of relevant data, benefit cost ratios were estimated at project preparation. The main benefits were to be derived from an increase in the use of preventive and public health services by the poor. Over the project period, use of services increase by 27 percent in the four provinces, as compared to an increase of 80 percent that would be necessary to reach the outcome target. This shortfall is likely to have reduced the estimated benefits of the project. Moreover, any increased use is likely to have been driven mainly by the introduction of a poverty-targeted health assistance benefit that essentially eliminated direct cost barriers to 47 use for a large number of poor households; and by a steady increase in geographical service coverage that was largely independent of the project. The block grant mechanism - the bulk of spending projected under the project - appears to have collapsed, with only some 12 percent of resources spent. In addition, noting that few of the outputs were attainted, or only partly attained, this further reduces the efficiency of resource use under the project. 4.29 Project efficiency is rated negligible. Ratings OUTCOME 4.30 The outcome is rated unsatisfactory. 4.31 The relevance of the project development objective is rated high, as it directly addressed a key policy variable - decentralization - that featured prominently in the country's development strategy and in the Bank's country assistance strategy. In addition, in including a particular focus on workforce management and on measures to enhance the quality of the workforce, it addressed an important input into a well-functioning health care system. Relevance of design is rated modest. While the design supported decentralization, it was introduced prematurely into an environment where decentralization still was finding its form; it may have been too ambitious; it provided insufficient safeguards against political and capacity risks; and the results framework provided an incomplete basis for monitoring project implementation and outcomes. This is in turn reflected in the ratings for efficacy - modest in both cases. None of the objectives were achieved - in some instances, objective indicators were partly achieved, but in most instances this was not the case. With regard to the first objective, where health outcomes were measured, while progress occurred, none of the targets actually were achieved. In the two other cases, there were no outcome indicators to measure achievement of the objectives. Efficiency is rated negligible, as the project fell far short of achieving its main efficiency target and had mixed results in reaching output targets. 4.32 Risk to Development Outcome 4.33 As in the case of the two other projects, the risk to the development outcome for HWS was rated moderate at appraisal and significant in the ICR. The risk to the development outcomes was both political and institutional. While the necessary legislation for decentralization had been put into place, the actual form decentralization would take in terms of the roles of the three government levels and the sharing of functions remained undefined. Here, the risk was the same as in PHP1 and PHP2, that project approaches and that the knowledge that was transmitted through the project would not be sustainable as the more detailed legislation took shape. In fact, decisions in key areas still have not been taken. 4.34 Over the longer term, the situation did change in two areas. Evidence-based teaching of medical students would spread, and licensing, certification, and registration of health professionals and accreditation of facilities would be pursued as a national concern, including 48 in a recent Bank project.9 And the effects of capacity building at the district level in planning and budgeting, and in engaging civil society, would also take root, albeit within constraints on resource management set by the central level. 4.35 In the absence of a process of decentralization as perceived by the Bank project in its design, the risk to the development outcome is rated significant. BANK PERFORMANCE 4.36 Quality at Entry. (1) Project design was consistent with sector priorities - decentralization and better service quality, and the approach focused on major challenges of decentralization. The project drew on the same analytical work as the two earlier projects. (2) Technical design was sound - institution building at province and district levels followed the approach of the two preceding projects, and the interventions intended for strengthening the health workforce were appropriate generally and adapted to identified needs in a decentralized environment; (3) Similarly to the two other projects, institutional arrangements were introduced early on to allow quick project startup. 4.37 Quality of entry was weak in a number of areas: project scope; risk assessment; monitoring and evaluation arrangements; other implementation arrangements. (1) The project scope appears too ambitious compared to client capacity, as reflected in the number of outputs that were only partly achieved or not achieved or implemented at all. (2) The introduction of a health manpower component - while addressing a key issue in the health system - falls awkwardly into the development model that the project otherwise adopted from PHP1 and PHP2. It is not apparent how it was integrated into the decentralization focus of the project or how it supported it. (3) Stakeholder analysis may have been insufficient - it did not capture client absorptive capacity, nor the uncertainties that surrounded decentralization; both might have been better assessed by closer exchange of experiences with the PHP1 and PHP2 teams. (4) This disconnect is, in turn, reflected in the risk rating "modest." (5) In terms of implementation design, the problem with transfer of grant funds that plagued all three projects, was not addressed, leading to subsequent implementation problems. (6) An insufficient results framework made it difficult to adequately monitor the project or assess outcomes. 4.38 Quality at entry is rated unsatisfactory. 4.39 Quality of Supervision. Supervision was dogged by some of the problems raised in Quality of Entry above. The uncertainties about the government's intentions for the design of decentralization that had affected the other projects appeared to have an even stronger effect on HWS. It frustrated supervision efforts and ending in the cancellation of some $49 million of the Bank Loan and IDA Credit. Monitoring efforts suffered from absence of data, and a midterm change of course, introducing outside assistance to develop a monitoring and evaluation strategy, was too late to be very meaningful. The project was also included in the efforts to deal with the corruption that appeared under PHP2, and the adjustments, especially 9 Health Professional Education Quality Improvement Project (P11334; 2010) 49 the shift in overall coordination, affected this project as well as the two others. The marginal impact of that, in terms of technical implementation was insignificant. 4.40 Quality of supervision is rated moderately unsatisfactory. 4.41 Overall Bank performance is rated unsatisfactory. BORROWER PERFORMANCE 4.42 Government performance. No substantive progress was made on decentralization (sharing of functions) during implementation of the project. Regulations were passed concerning the sharing of functions, but the division of responsibilities remained unclear. Cumbersome procedures for the disbursement of grant funds continued, causing severe delays in the flow of funds, and finally leading to cancellation of some Credit funds, as well as a significant shortfall in the disbursement of district grants. A_Bank proposal to extend the project to better achieve objectives did not get government support. 4.43 Government performance is rated unsatisfactory. IMPLEMENTING AGENCY PERFORMANCE * The Ministry of Health. The shortcomings noted in the previous projects were also present in this project, notably an unclear mandate, limited capacity, and difficulties in project management. Outside the project framework, the ministry did initiate technical support activities, focusing on weaker districts. At the interagency level, the workforce development component was constrained by ineffective coordination between implementing agencies (Ministry of Health, Ministry of National Education, and Indonesian Medical Association) that limited the use of workforce policies at sub- national levels. * Provinces and districts. As was the case in PHP1 and PHP2, implementation performance at province and district levels was mixed. In no instances does it appear to have reached levels comparable to those in Yogyakarta and West Java. In this project, capacity building in planning and investment was particularly hampered by slow disbursement of grant funds. Partly as a consequence, districts were more likely to apply such funds to meeting recurrent spending needs, rather than for investments, either in new facilities or in equipment for primary care. 4.44 Implementing agency performance is rated unsatisfactory. 4.45 While overall borrower performance is rated unsatisfactory on the basis of the above, the following should be noted. Project design and project objectives were largely driven by Bank assumptions about decentralization that had little basis in regulation and subsequently proved to be wrong. 50 MONITORING AND EVALUATION 4.46 Design. The project documents offered little guidance on the collection and analysis of information on institutional development, beyond the routine monitoring that was undertaken by project coordinating teams. The results framework was too vague to offer much scope for that. Health outcome indicators were an exception, and for them baseline data had been defined and targets set, with data drawn from official mortality data and Demographic Health Survey data. 4.47 Implementation. The implementation of M&E focused on the delivery of inputs and outputs; beyond MDG-related health outcome indicators, there is no documentation available suggesting that attention had been devoted to outcome monitoring. After midterm review, an international consultant was recruited by the Bank to develop an M&E strategy. Based on recommendations of the consultant, performance indicators were revised and information on them was collected for the final project evaluation. The indicators still were output indicators. 4.48 Use of Data. Beyond the assessment of investment proposals, data were not used for decision-making. 4.49 Monitoring and evaluation is rated negligible. Summary Conclusions from the Three Projects CRITICAL ASSUMPTIONS 4.50 All three projects were based on a number of critical assumptions that proved untenable: a) The first and overriding assumption was that the government, particularly the MOH, supported the model of decentralization that the three projects were piloting. This was not an unreasonable assumption: framework legislation for decentralization had been passed, and the three projects, especially PHPI and PHP2, would be introducing functions and roles at the three government levels that could be considered appropriate for a decentralized environment. And these matters had been discussed extensively by the Bank and the health authorities during the previous decade. Still, defacto responsibilities remained uncertain, and the MOH would be hesitant about reshaping its role and the ability of provinces and districts to take on the roles envisaged for them. b) This uncertainty about roles and responsibilities at the three levels of government weakened other critical assumptions: the coordinating role of provinces and, more generally, the relationship between provinces and districts that was envisaged under the projects; the allocation and use of human and financial resources across the system; and an effective stewardship and advisory role for the MOH. c) A "technical" assumption was that the grant funds allocated to the sector would be consistent, sufficient, and predictably disbursed. This turned out not to be the case - 51 only some 70 percent of the $1,2 billion allocated for this purpose was distributed, weakening the impact of the substantial grant resources that were supposed to underpin the planning and investment process at district levels. Moreover, there is little clarity about the effectiveness of the use of these funds. DID THE PROJECTS MAKE A MEASURABLE DIFFERENCE? 4.51 Overall, the projects had only a modest impact on decentralization. They did build capacity for planning and budgeting in participating districts, which in some instances has been expanded upon. Their role at province levels has been similar: while province functions have been circumscribed, in some instances the model developed by the projects has come to be used in a coordinating role in support of districts. In the MOH, achievements were modest. Finally, in terms of outcomes, there is little difference between participating districts and the rest of the country. 4.52 Could the projects have been reconfigured to better support the process, such as it was? All three projects contained useful elements focused on building up policy and planning capacity at the district level. However, as the situation evolved, this capacity building may have been overwhelmed by the ambitious investment program that was supported under the projects, and which appears to have borne little relationship to any recurrent financing burden that arose from such investments. Projects of more modest scale, focused on capacity building in policy planning, budgeting, and investment management at the district level might have better prepared province and districts authorities to position themselves in the fluid environment that was created by decentralization legislation. In fact, it would turn out that districts did react opportunistically to the uncertain regulatory environment, often initiating investments that would turn out to be unaffordable and reduce the quality of services not only in new facilities, but also in existing ones. 4.53 Outcomes. The following charts compare service inputs and health outcomes in the nine districts targeted by the three projects to national trends. The districts are also compared among each other. In examining the charts, it should be kept in mind that the World Bank was not the only agency active at strengthening performance at province and district levels. The Asian Development Bank was managing a large district-level program at the same time, covering some 20 districts; and the U.S. Agency for International Development was supporting maternal and child health in another 25 districts. While none of these overlapped with Bank-targeted districts, they did coincide in some instances at the province level. 52 Figure 4.1. Indonesia - Trends in Infant and Under-5 Mortality Rates, Various Years Infant mortality/1000 Infant Mortality 80 Project provinces 70 60 _____________________ 80 50 60 40 30 40 m1997 20 20 m2002 10 0 002007 1994 1997 2002 2007 4 4 m Province a National Under 5 Mortality/1000 Under 5 Mortality 100 Project Provinces 80 100 60 80 60 60 40 40 m1997 20 *t20 12002 0 . .2007 1997 2002 2007 m Province 0 National Source: Indonesia District Health Survey 1996, 2003, 2007 53 Figure 4.2. Indonesia - Trends in Selected Indicators of Service Provision, Selected Years Service provision - Birth Birth attendance attendance Project provinces 80 100 65 - 80 60 50 - 40 m1997 20 -2002 0 m2006 20 Nb e 1992 1995 2000 2003 2006 2009 o Full immunizations Full immunizations (BCG, DPT, Polio, Measles) Project provinces 100 100 80 80 60 6 0 m1997 40 1997 2003 2007 o so s a Province E National Source: Indonesia District Health Survey 1996, 2003, 2007 54 Table 4.4. Rates of Change (annual percentage rates) in Selected Health Services and Outcome Indicators, 1997-2007 Birth Attendance Infant Mortality Full Immunization Under 5 Mortality NATIONAL PROVINCES NATIONAL PROVINCES NATIONAL PROVINCES NATIONAL PROVINCES 1997-2007 7.8 5.1 3.8 3.1 -1.1 2.5 2.5 3.2 1997-2002 10.7 8.0 3.5 4.4 -4.0 -2.8 5.1 5.2 2003-2007 2.7 1.1 5.2 1.8 3.4 10.0 -1.2 0.9 Source: Indonesia District Health Survey 1996, 2003, 2007 4.54 The Figures 4.1 and 4.2, and Table 4.4 examine trends in selected indicators during the period 1997-2007. They draw on available data, notably from the Indonesia District Health Surveys of various years over that period. Detailed data for subsequent periods was not available at the time of the review. The period predates and covers most of the time of implementation of the three projects. They indicate the following: Birth attendance and infant mortality: * Over the years 1997 to 2007, birth attendance at the national level increased at an annual rate of 7.8 percent, rising from 43 percent to 73 percent of attended births. Prior to the political changes, the increase was 10.7 percent per year, but dropped to 2.7 percent per year during the subsequent period. This may reflect a number of things: increased coverage; administrative uncertainties created by decentralization; and the effects of a de-emphasis of village-level services after 2000. The latter is widely regarded as an effort to break with earlier tradition. The village focus has subsequently been restored with the introduction of a "healthy village" program that was introduced around 2005. * For most of the period, birth attendance has been higher in the project provinces as a whole compared to the national averages. This mainly reflects a higher starting point. The evolution of birth attendance in the project provinces has followed the national trend, albeit at a lower rate of growth, and birth attendance at the national level has now (in 2007) caught up with the project provinces. * In the absence of comparable maternal mortality figures, the PPAR uses infant mortality as a proxy outcome indicator for birth attendance. Both national and province level start out at roughly similar level of infant mortality - 52 and 54 per 1,000, respectively. Prior to the projects, infant mortality fell faster in the project provinces; during the 2003-2007 period, national level performance was significantly better, and by 2007, infant mortality at the national level was at 34 per 1,000, and at 39 per 1,000 in the project provinces. Arguably, a consistently stronger performance in attended births at the national level could support this trend. 55 Immunizations and under 5 mortality * Nationally, full immunization coverage dropped throughout the period, from 55 to 50 percent. This includes a sharp drop between 1997 and 2002 of some 4 percent per year, and a subsequent rebound. The project provinces followed a similar trend, but consistently remained above national coverage rates: in 1997 it was 57 percent and by 2007 it had risen to 70 percent. It appears that the immunization program was badly affected by the political transition and the uncertainties that may have surrounded responsibilities for vaccination programs - centralized or decentralized - at that time. It is worth noting that in 1997, at the start of the data set, six of the nine provinces had full immunization coverage lower than the national average; in 2007 all were above the national average. * Under 5 mortality has consistently been lower at the national level, even with a slight increase between 2003 and 2007. Over the full period, they have declined from 58 to 45 per 1,000, falling by some 2.5 percent. Although under 5 mortality rates have consistently fallen in the project provinces, they have remained higher than the national average: in 1997, the rate was at 72 per 1,000; and by 2007, it had fallen to 51 per 1,000. 4.55 The Charts and the Table offer little evidence that the pilot projects were making a measurable difference in outcomes when compared to performance nationally: performance was nothing out of the ordinary, maybe even slightly worse. A number of factors may have intervened to blur the distinctions and influence results. Comparison between performance in project provinces and other provinces (or nationally) is difficult, as extensive support by other donors in other provinces and districts had objectives that were likely to similarly influence the same indicators. Moreover, high errors are attached to the data because of the small sample that underlies the DHS surveys; and because of infrequent and inefficient collection and reporting of service statistics, as district reporting to provinces and the center became voluntary. 4.56 Decentralization at district and province level. While achievement of project objectives as measured by efficacy was modest at best in all three projects, the projects did introduce better planning and budgeting methods that over time appear to have contributed to laying the basis for understanding the need for and the use of information for improved performance both at province and district levels. Recipient provinces and districts have built on this transfer of knowledge to establish better systems, and they have served as models for other provinces and districts in developing their own systems. This has been the case especially in West Jawa, Yogyakarta, and East Kalimantan, where authorities early on were able to combine new knowledge with existing local knowledge bases. In other provinces, results have been more mixed, largely depending on local capacity to build on project inputs and examples. That said, the project organizations that originally were set up to institutionalize better planning and budgeting (JHC, TRT, and TF) largely disappeared after the projects ended. 4.57 Decentralization and the Ministry of Health. Where the projects have been least successful is in the MOH. This may in part be due to the cautious attitude in the ministry to 56 decentralization, including the absence of regulations and persistent doubts about the capacity of districts to take on new responsibilities. It is reflected in the disappointing results of the institution building for decentralization in the ministry. It is also reflected in the absence of instruments to oversee performance in the sector. Information flows between the districts as the key actors in service provision and the MOH manifest gaps in quantity, quality, and timeliness of information, as district accountability shifted to local parliaments and civil society. This negatively affected both central planning and allocation of transfer resources, and perceived relevance of national policy for district authorities. The central ministry faces challenges in encouraging districts to support national priorities, and it has been reluctant to use the leverage it still has through its control over financial and human resources to influence district planning. 5. Lessons 5.1 Policy reform in a volatile environment. When undertaking extensive policy reforms in a rapidly changing political and institutional environment, challenges and risks posed by traditional ways of doing business are likely to be formidable; and even thorough analytical work and sound technical designs may not be enough to achieve good results. The three projects are cases in point. They were all based on strong analytical work and best practice. Still, the results were unsatisfactory. This may well reflect that insufficient attention was being paid to the political economy environment. In Indonesia, reluctance to introduce accompanying regulations to new laws was an indicator that new institutional arrangements had yet to mature; and that government was not yet ready to firm up its options for defining new responsibilities. Such risks need to be recognized and planned for. 5.2 Monitoring and evaluation. The Bank can support high-risk/high potential projects, but a prerequisite must be that strong M&E in place; and that it incorporates appropriate technical as well as political economy indicators; includes a schedule for rigorous assessment of progress and results; and allows lessons to be drawn for subsequent projects. 5.3 Goals and project organization need to be well understood and supported by the client for effective M&E and learning. Successful introduction of new institutions requires that clients - in this case MOH and sub-national governments - understand the changes that are being sought, believe they are feasible, support them, and are ready to pursue them. This requires collaborative project development and continuous awareness-building of the project among participants to ensure that the client is able to understand what is going on, support it, and ultimately own it. The importance of this is highlighted by discussions with representative clients during project completion workshops and during interviews for this PPAR. There was uncertainty about project goals and institutions at all three levels of government. An insufficiently elaborated results framework and related monitoring and evaluation arrangements underscored the vagueness of the activities 5.4 An objective assessment of the client's technical capabilities to take on new roles is a prerequisite for success. The design and timing of inputs - the roadmap for implementation - must be calibrated to client absorptive capacity, especially where there may be multiple clients of varying levels of maturity. In this case, capacity was strong in some localities, but limitations in others made project targets much less feasible. Design 57 needs to be calibrated to the environment, and this may require adjustments to design, capacity building, and resource planning. 5.5 Conventional government processes that enter into project design have to be functioning well, geared to facilitating implementation, and amenable to adjustment if necessary, especially when the emphasis is on piloting new methods. This is particularly important when complex new approaches are being tested. The experience under these three projects provides an illustration. Slow budget processes caused major delays in the district- level planning and implementation process, disturbing the learning experience. Fiduciary controls proved inadequate, delaying purchasing processes, and in one instant causing cancellation of resources for one project (PHP2). 5.6 Sequencing of projects. All three projects were piloting new complex reforms in an unsettled environment. All three were closely sequenced, offering limited opportunity for transmitting learning from one project to another. Projects introducing new institutions must be seen as opportunities for learning and improving, and sequencing should have that in mind. 5.7 Additional lessons more specific to these decentralization projects may be drawn from the summary of two learning workshops provided in Attachment 1. The workshops were conducted at the province level in East and West Kalimantan and one at the central level, and may serve as a basis for drawing broader lessons in designing projects.  59 References World Bank Documents World Bank. 2000. "Health Strategy in a Post-Crisis Decentralizing Indonesia." Report 21318-IND. Washington, DC. . 2003. "Decentralization in Indonesia: A Regional Public Expenditure Review." Overview Report. Report No. 26919_IND. Washington, DC. . 2007. "Spending for Development: Making the Most of Indonesia's New Opportunities." Indonesia Public Expenditure Review. Jakarta: World Bank Office. . 2008. "Investing in Indonesia's Health: Challenges and Opportunities for Future Public Spending." Health Public Expenditure Review 2008. Jakarta: World Bank Office . 2009a. "Giving More Weight to Health: Assessing Fiscal Space for Health in Indonesia." Report No. 46847-ID. Washington, DC. 2009b. "Indonesia's Doctors, Midwives and Nurses; Current Stock, Increasing Needs, Future Challenges and Options." Jakarta: World Bank Office . 2012. "Health Equity and Financial Protection Data Sheet. East Asia and Pacific." Washington DC. Project-related World Bank Documents IEG. 2008a. Indonesia Id-provincial Health II Implementation Completion Review (P049539). Washington, DC: World Bank. 2008b. Indonesia Provincial Health I Implementation Completion Review (P049545). Washington, DC: World Bank. 2010. Health Workforce & Services (Phase 3) Implementation Completion Review (P073772). Washington, DC: World Bank. World Bank. 2000. "Project Appraisal Document on a Proposed Credit in the amount of SDR 28.6 million (US$ 38 million) to the Republic of Indonesia for Provincial Health I." Report 20423-IND. Washington, DC. . 2001. "Project Appraisal Document on a Proposed Credit in the amount of SDR 31.4 million (US$ 40 million) and Loan of US$ 63.2 million to the Republic of Indonesia for ID-Provincial Health Ii." Report 22051-IND. Washington, DC. 2003. "Project Appraisal Document on a Proposed Credit in the Amount of US$ 74.5 million equivalent and Loan of US$ 31.1 million to the Republic of Indonesia for a Health Workforce and Services Project. " Report no. 25294. Washington, DC. . 2008a. "Implementation Completion and Results Report (IBRD-46290 IDA-35370) on a Loan in the Amount of US$ 63.2 million and a Credit in the amount of SDR 31.4 million (US$ 40 million) to the Republic of Indonesia for a Second Provincial Health Project." Report no. ICR668. Washington, DC. . 2008b. "Implementation Completion and Results Report (IDA-33810) on a Credit in the amount of SDR 28.6 million (US$ 38 million) to the Republic of Indonesia for a Provincial Health Project." Report no. ICR786. Washington DC. . 2009. "Implementation Completion and Results Report (IBRD-47020/ IDA-37840) on a Proposed Credit in the Amount of US$ 74.5 million equivalent and Loan of US$ 31.1 million to the Republic of Indonesia for a Health Workforce and Services Project." Report no. ICRI 1111. Washington, DC. 60 Articles Bossert, Thomas. 1998." Analyzing the Decentralization of Health Systems in Developing Countries: Decision Space, Innovation and Performance." Social Science and Medicine 47(10): 1513-1527. Heywood, Peter, and Choi, Yoonyoung. 2010. "Health Systems Performance at the District Level in Indonesia after Decentralization." BMC International Health and Human Rights 10(3). Heywood, Peter, and Harahap, Nida. 2009. "Public funding of health at the district level in Indonesia after decentralization - sources, flows and contradictions." Health Research and Policy Systems 7(5). Hofman, Bert and Kaiser, Kai. 2002. "The Making of the Big Bang and its Aftermath." Presentation at Georgia State University Conference. Atlanta, Georgia. Kristiansen, Stein and Santoso, Purwo. 2006. "Surviving Decentralization? Impacts of regional autonomy on health service provision in Indonesia." Health Policy 77: 247-259. Kruse, loana, Pradhan, Menno, and Robert Sparrow. 2009. "Health Spending and Decentralization in Indonesia." Unpublished draft. Lieberman, Samuel S., Capuno, Joseph J., and Hoang Van Minh. 2005. "Decentralizing Health: Lessons from Indonesia, the Philippines, and Vietnam." Discussion Paper No. 0408, WB-Vietnam, University of the Philippines and Hanoi Medical School. Mardiati et al, WHO. 2004. "Synthesis study of District Health Accounts." Jakarta, Indonesia. Sandi, Andi Ant. T.T, Vivi Lignawati. 2004. "Desentralisasi dan Perkembangan Peraturan Perundang-undangan di Bidang Kesehatan: Sebuah Evaluasi Normatif. Makalah Seminar Perjalanan 3 Tahun Desentralisasi Kesehatan di Indonesia: Apa yang sudah dicapai dan apa yang belum?" Tanggal 17-19 Maret 2004 di Fakultas Kedokteran, Universitas Gadjah Mada. www.desentralisasi-kesehatan.net/id/ Simantupang, Rntanida Renata. 2009. "Evaluation of Decentralization Outcomes in Indonesia: Analysis of Health and Education." Dissertation, Department of Economics: Georgia State University. Atlanta, Georgia. Thabrany, Hasbullah . 2006. "Human Resources in Decentralized Health Systems in Indonesia: Challenges for Equity." Regional Health Forum 10(1): 77-8 8. Tilden, Robert, Noor, Nasry Nur, Ascobat Ghani, and James Sonnemann. 2006. "The effect of Decentralization on the health status and health care utilization patterns in Indonesia." TA 3579 INO. Ministry of Health, Republic of Indonesia, Asian Development Bank, EPOS. Gizi Indonesia 29 (2). Jakarta. Government Reports Government of Indonesia, ADB, AusAID, GtZ, WHO, World Bank. 2008. Background paper on the Indonesian Health System in Support of the Government ofIndonesia Health Sector Review. Mimeograph. Ministry of National Development Planning. 2009. Roadmap for the Achievement of the Millenium Development Goals in Indonesia. Ministry of national Development Planning/Development Planning Agency. . 2011. Report on the Achievement of the Millenium Development Goals in Indonesia. Ministry of National Development Planning/Development Planning Agency. 61 Data sources Health Indicators 1995-2010. Statistics Indonesia, Jakarta, Indonesia Indonesia Demographic and Health Survey 1997, 2002-2003, 2007. Statistics Indonesia, Jakarta, Indonesia. Profil Kesehatan Indonesia, 2002-20 10. Departemen Kesehatan, Jakarta, Indonesia (various years). National Health Accounts, Indonesia, 2002, 2004  63 ANNEX A Annex A. Basic Data Sheet for Indonesia Provincial Health I Project Name (P049545) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 78.63 113.04* 143.76 Loan amount 38.00 30.50 80.26 Cancellation 0.00 10.65 0.00 * Including 2 year District Grant extension Cumulative Estimated and Actual Disbursements FY00 FYO1 FY02 FY03 FY04 FY05 FY06 FY07 FY08 Appraisal estimate (US$M) 3.2 10.7 19.2 27.1 33.2 38.0 38.0 38.0 38.0 Actual (US$M) 0.0 1.0 3.4 11.9 16.7 21.7 28.4 29.8 30.5 Actual as % of appraisal 0.0 9.3 27.8 43.9 50.3 57.1 74.7 78.4 80.2 Date of final disbursement: 12/02/2007 Project Dates Original Actual Initiating memorandum 07/27/1998 08/14/1998 Negotiations 12/20/1999 05/02/2000 Board approval 02/01/2000 06/15/2000 Effectiveness 08/21/2000 08/21/2000 Closing date 06/30/2006 12/31/2007 ANNEX A 64 Staff Time and Cost Stage of Project Cycle Staff Time and Cost (Bank Budget Only) US$ Thousands (including Lending travel and consultants costs) FY97 7.80 FY98 33.64 FY99 93.58 Total: 135.02 Supervision/ICR FY00 57 102.48 FYO1 22 32.46 FY02 32 74.43 FY03 14 57.82 FY04 30 126.69 FY05 25 76.11 FY06 21 46.91 FY07 19 53.71 FY08 20 45.03 Total: 240 615.64 Task Team Members Names Title Unit Lending Samuel S. Liberman Lead Economist EASHD Anthony Toft Chief Counsel LEGES Puti Marzoeki Senior Health Specialist EASHD William Hardi Consultant EAPCO Yogana Prasta Operation Adviser EACIF Naseer Ahmad Rana Adviser SARSQ Novira Kusdarti Asra Financial Management Specialist EAPCO Benedicta R. Sembodo Program Assistant EACIF Supervision/ICR Novira Kusdarti Asra Financial Management Specialist EAPCO Jed Friedman Economist DECRG William Hardi Consultant EAPCO Pandu Harimurti Health Specialist EASHD Adrian Hayes Consultant EASHD 65 ANNEX A Names Title Unit Peter F. Heywood Consultant EASHD Edhie Santosa Rahmat Consultant EASHD Claudia Rokx Lead Health Specialist EASHD Imad Saleh Lead Procurement Specialist EAPCO Agus Sasmito Consultant EASHD Anita Kentjanawati Tuwo Procurement Specialist EAPCO Benedicta R. Sembodo Program Assistant EACIF Ryma F.J. Aguw Team Assistant EACI ANNEX B 66 Annex B. Basic Data Sheet for Indonesia Provincial Health II Project (P049539) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 895.81 783.62 87.47 Loan amount 103.20 56.98 55.21 Cancellation 0.00 50.72 0.00 Cumulative Estimated and Actual Disbursements FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 Appraisal estimate (US$M) 1.5 17.3 42.9 69.2 92.0 103.2 103.2 103.2 103.2 Actual (US$M) 0.0 0.6 4.6 8.3 28.8 35.3 75.2 56.9 56.9 Actual as % of appraisal 0.0 3.4 10.7 11.9 31.3 34.2 72.8 55.1 55.1 Date of final disbursement: 01/28/2009 Project Dates Original Actual Initiating memorandum 05/01/2000 12/21/2000 Negotiations 03/01/2000 05/15/2001 Board approval 03/28/2000 06/26/2001 Signing 08/03/2001 08/03/2001 Effectiveness 04/24/2002 04/24/2002 Closing date 06/30/2007 31/07/2009 67 ANNEX B Staff Time and Cost Stage of Project Cycle Staff Time and Cost (Bank Budget Only) US$ Thousands (including Lending travel and consultants costs) FY99 0.18 FY00 2 30.69 FYO1 41 70.27 Total: 43 101.14 Supervision/ICR FY02 15 53.74 FY03 15 51.19 FY04 23 68.16 FY05 31 78.05 FY06 31 99.42 FY07 21 75.58 FY08 2 3.33 Total: 138 429.47 Task Team Members Names Title Unit Lending Samuel S. Lieberman Lead Economist EASHD Puti Marzoeki Senior Health Specialist EASHD William Hardi Consultant EAPCO Novira Kusdarti Asra Financial Management Specialist EAPCO Yogana Prasta Operations Adviser EACIF Anthony Toft Chief Counsel LEGES Benedicta R. Sembodo Program Assistant EACIF Supervision/ICR Novira Kusdarti Asra Financial Management Specialist EAPCO Jed Friedman Economist DECRG Pandu Harimurti Operations Officer EASHD Peter F. Heywood Consultant EASHD Puti Marzoeki Senior Health Specialist EASHD Edhie Santosa Rahmat Consultant EASHD Claudia Rokx Lead Health Specialist EASHD Anita Kentjanawati Tuwo Procurement Specialist EAPCO ANNEX B 68 Names Title Unit Benedicta R. Sembodo Program Assistant EACIF Ryma F.J. Aguw Team Assistant EACIF William Hardi Consultant EAPCO Yogana Prasta Operations Adviser EACIF Agus Sasmito Consultant EASHD 69 ANNEX C Annex C. Basic Data Sheet for Indonesia Health Workforce and Service Project (P073772) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 429.37 391.60 91.20 Borrower 323.77 335.89 103.74 Loan amount (IBRD) 31.10 0.31 0.99 Loan amount (IDA) 78.99 55.40 70.13 Cancellation (IBRD) 0.00 30.79 0.00 Cancellation (IDA) 0.00 17.17 0.00 Cumulative Estimated and Actual Disbursements FY04 FY05 FY06 FY07 FY08 FY09 Appraisal estimate (US$M) 16.00 44.60 71.40 91.40 102.10 105.60 Actual (US$M) 4.00 12.71 20.51 47.49 48.85 55.40 Actual as % of appraisal 25.00 28.49 28.72 51.95 47.84 52.46 Date of final disbursement: 02/25/2009 Project Dates Original Actual Initiating memorandum 10/02/2001 10/02/2001 Negotiations 05/01/2003 05/01/2003 Board approval 04/11/2002 06/12/2003 Signing 09/30/2003 10/01/2003 Effectiveness 12/30/2003 12/30/2003 Closing date 12/31/2008 12/31/2008 ANNEX C 70 Staff Time and Cost Stage of Project Cycle Staff Time and Cost (Bank Budget Only) US$ Thousands (including Lending No. of staff weeks U$Tosns(nldn travel and consultants costs) FY97 7.80 FY98 33.64 FY99 93.58 FY00 57 102.48 FYO1 22 32.46 FY02 32 74.43 Total: 111 344.39 Supervision/ICR FY03 14 57.82 FY04 30 126.69 FY05 25 76.11 FY06 21 46.91 FY07 19 53.71 FY08 20 45.03 Total: 129 406.27 Task Team Members Names Title Unit Lending Samuel S. Liberman Lead Economist EASHD Anthony Toft Chief Counsel LEGES Put Marzoeki Senior Health Specialist EASHD William Hardi Consultant EAPCO Yogana Prasta Operation Adviser EACIF Naseer Ahmad Rana Adviser SARSQ Novira Kusdarti Asra Financial Management Specialist EAPCO Benedicta R. Sembodo Program Assistant EACIF Supervision/ICR Puti Marzoeki Senior Health Specialist EASHD Novira Kusdarti Asra Financial Management Specialist EAPCO Jed Friedman Economist DECRG William Hardi Consultant EAPCO 71 ANNEX C Names Title Unit Pandu Harimurti Health Specialist EASHD Adrian Hayes Consultant EASHD Peter F. Heywood Consultant EASHD Edhie Santosa Rahmat Consultant EASHD Claudia Rokx Lead Health Specialist EASHD Imad Saleh Lead Procurement Specialist EAPCO Agus Sasmito Consultant EASHD Anita Kentjanawati Tuwo Procurement Specialist EAPCO Ryma F.J. Aguw Team Assistant EACIF ANNEX D 72 Annex D. Results of Workshops on Lessons Learned from the Decentralization Projects The following summarizes the results of two workshops conducted in East and West Kalimantan, and the MOH, respectively, on lessons from the Bank's HWSP project. Due to the similarity of design of the three projects, the lessons should be applicable to all three projects. Project design * Province and district stakeholders saw the design as too ambitious. However, it did have the advantage of providing equal opportunity for all three administrative levels to build capacity in the new decentralization era; * Stakeholders welcomed the opportunity to conduct bottom-up planning and many believed that establishing the capacity to conduct evidence-based planning was a major achievement of the project; * The grants component channeled was administratively ambitious, as it involved a large number of districts receiving fund allocations; * Participants at the central level felt that the MOH components would have been stronger with a more effective MOH involvement during project preparation; * Complex projects like HWS require a lot of learning and adaptation, particularly during the first years of implementation, and a longer implementation period is warranted * There was consensus among the provinces and the districts that technical review teams at the province level that reviewed plans and projects, had usefully contributed to strengthening the planning process and improved province-district collaboration Project objectives and indicators * Workshop participants at all levels agreed that clear objectives and indicators are important for guiding project implementation and helps to focus the operation; * When asked about project objectives, province participants were most familiar with the objective to improve the capacity of human resources for health (HRH); only a few stakeholders were aware of the project objectives as written in the legal documents and the PAD; * Continuous "socialization" of the project was important, given the high staff turnover and complexity; * Central level participants felt that the objectives of the central components were not well defined; * Lack of awareness of project indicators and their importance for measuring project performance contributed to lack of focus during implementation. 73 ANNEX E Annex E. List of Persons Met World Bank Novira Asra - Senior Financial Management Specialist Enis Baris - TTL for HWSP Mae Chu Chang - Human Development officer, Jakarta Darren Dorkin - Human development manager, Jakarta Samuel Lieberman - Lead Economist, TTL for PHP1 and PHP2, Pandu Harimurti - Health specialist, Jakarta Puti Marzoeki - Senior Health Specialist, TTL for PHI, PH2 and HWSP Yogana Prasta - Operations Advisor Fadia Saadah - Sector leader Lyndsay Thomas - Community Liaison, Jakarta William Wallace - Senior Advisor, Jakarta Ministry of Health Dr. Madiono - Former Head of Planning and Budgeting Dr. Isti Ratnaningsih - Former Head, Decentralization Group Imam Subekti - Former Head of International Affairs Dr. Untung Sutarjo - Chairman, Health and Human Resources; former Head of Policy Unit Dr. Siamet Riyadi Yuwono - Director General of Nutrition and MCH Dr. Widiyarti - Senior Health Planner Dr. Setiawan Soeparan - Director General of pharmacy; former member of project management team Dr. Doti Indrasanto = former Head of Data and Information Dr. D.K. Wirakamboja - former Head of Program Evaluation Ministry of National Education Prof. Djoko Santoso - Director General of Higher Education, Project Director for Health Professional Education Quality Project State Planning Agency Dr. Arum Atmawikarta - MDG Secretariat Vivi Yulaswati - Director of Social Services Alan Prouty - Project manager, MDG Roadmap, Bappenas Universities and Research Institutes Professor Ascobat Gani - Head of Health Economics Dpt, University of Indonesia Dr. Laksono - Head of Economics Dpt, Gadjah Mada University Sri Budiyati - Senior Researcher, SMERU (main independent social research agency) Others ANNEX E 74 Dr. Nida Harahap - Health policy researcher David Deziel - Head of Social Protection and Health; Hickling Corp., Jakarta Dr. Rachmid Untoro - Project manager, PHP1, PHP2, and HWSP Antarini Antojo - Former member of MOH project management unit Haryoko Wiharjo - consultant for PHP1 and PHP2 Rachel Cintron - Dpty Director Office of Health, USAID Harmain Harun - Senior Advisor, GtZ Dr. Rooswanti Soehamo - Health Advisor, ADB Jakarta Rabin Hattari - Public Finance Economist, ADB