Document of The World Bank Report No: ICR0000472 IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF-050152) ON A GRANT IN THE AMOUNT OF US$20.6 MILLION TO THE DEMOCRATIC REPUBLIC OF TIMOR-LESTE FOR A FUNDAMENTAL SCHOOL QUALITY PROJECT June 28, 2007 Human Development Sector Unit East Asia and Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective January 2007) Currency Unit = U.S. Dollar US$ 1.00 = US$ 1.00 US$ 1.00 = US$ 1.00 FISCAL YEAR [July 1 ­ June 31] ABBREVIATIONS AND ACRONYMS AusAID Australian Agency for International Development CMU Construction Management Unit ESRP Emergency School Readiness Project ETTA East Timor Transitional Administration FSQP Fundamental School Quality Project CITL CARE International Timor-Leste CFET Consolidated Fund for East Timor CNRT National Council of Timorese Resistance ECHO European Community Humanitarian Organization EP Escolas Primarias (Primary Schools) ESPR Emergency School Readiness Project ESSP Education Sector Support Project FSQP Fundamental School Quality Project GER Gross Enrollment Rate IFU Infrastructure Facilities Unit MoEC Ministry of Education and Culture NER Net Enrollment Rate NGO Non-government Organization PDO Project Development Objective PESP Primary Education Sector Support Project PMU Project Management Unit SSRP School System Revitalization Program TFET Trust Fund for East Timor SP-UPC Strategic Plan for Universal Primary Completion UNICEF United Nations Children's Fund UNTAET United Nations Transitional Administration in East Timor WHO World Health Organization Vice President: James W. Adams Country Director: Nigel Roberts Sector Manager: Christopher J. Thomas Project Team Leader: Jerry G. Strudwick ICR Team Leader: Jerry G. Strudwick TIMOR-LESTE Fundamental School Quality Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 3 3. Assessment of Outcomes............................................................................................ 7 4. Assessment of Risk to Development Outcome......................................................... 11 5. Assessment of Bank and Borrower Performance ..................................................... 11 6. Lessons Learned ....................................................................................................... 13 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 15 Annex 1. Project Costs and Financing.......................................................................... 16 Annex 2. Outputs by Component ................................................................................. 17 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 20 Annex 5. Beneficiary Survey Results........................................................................... 22 Annex 5. Beneficiary Survey Results........................................................................... 22 Annex 6. Stakeholder Workshop Report and Results................................................... 23 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR..................... 24 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders....................... 34 MAP IBRD33496 A. Basic Information TP-FUNDAMENTAL Country: Timor-Leste Project Name: SCHOOL QUALITY PROJECT Project ID: P072647 L/C/TF Number(s): MULT-50152 ICR Date: 06/28/2007 ICR Type: Core ICR Lending Instrument: SIL Borrower: E. TIMOR Original Total USD 20.6M Disbursed Amount: USD 20.4M Commitment: Environmental Category: B Implementing Agencies: Ministry of Education and Culture Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 05/15/2001 Effectiveness: Appraisal: 05/21/2001 Restructuring(s): Approval: 09/28/2001 Mid-term Review: 01/28/2003 Closing: 11/30/2003 12/31/2006 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Low or Negligible Bank Performance: Satisfactory Borrower Performance: Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory Quality of Supervision: Satisfactory Implementing Agency/Agencies: Satisfactory Overall Bank Overall Borrower Performance: Satisfactory Performance: Satisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Performance Indicators (if any) Rating Potential Problem Project Yes Quality at Entry None i at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 12 12 Primary education 44 44 Secondary education 44 44 Theme Code (Primary/Secondary) Conflict prevention and post-conflict reconstruction Secondary Secondary Education for all Primary Primary Education for the knowledge economy Primary Primary Participation and civic engagement Secondary Secondary E. Bank Staff Positions At ICR At Approval Vice President: James W. Adams Jemal-ud-din Kassum Country Director: Nigel Roberts Klaus Rohland Sector Manager: Christopher J. Thomas Maureen Law Project Team Leader: Jerry G. Strudwick Alfonso F. de Guzman ICR Team Leader: Jerry G. Strudwick ICR Primary Author: Robert L. McGough F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) 1. To support the Government's efforts to maintain the existing level of primary education enrollment (185,000 students, estimated 95% gross enrollment) and junior secondary enrollment (i.e., junior secondary education, 30,000 students, estimated 60% gross enrollment), with the possibility of increase due to return of refugees and growth of school-age population. 2. To continue to recover quality of primary and junior education by rehabilitating physical facilities, and by providing textbooks and instructional materials. ii Revised Project Development Objectives (as approved by original approving authority) Project Development Objectives were not revised. (a) PDO Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Revised Achieved at approval Target Completion or documents) Values Target Years Indicator 1 : Number of district sites where construction of Escolas Basicas is completed Value quantitative or 0 13 4 4 completed Qualitative) Date achieved 10/29/2001 11/30/2003 07/27/2004 12/31/2006 Comments (incl. % achievement) Indicator 2 : Number of Primary Schools upgraded to Fundamental Quality standard 74 new schools constructed; 23 schools refurbished. 13 site works 35 new completed for schools to be Value schools plus constructed by quantitative or 0 greater than 60 rehabilitation others; 72 schools Qualitative) of about 84 rural primary connected to shools existing water supplies; 2 deep bore wells provided in Dili; 43 ground water tanks constructed. Date achieved 10/29/2001 11/30/2003 07/27/2004 12/31/2006 Comments Project completed 102 schools of the revised target of 123 schools ­ 13 were (incl. % completed from other funding sources and the MoEC cancelled plans for 15 achievement) proposed project schools. Indicator 3 : Increase of enrollment in junior secondary over year 2000 Value At lease 2,600 Enrollment is quantitative or Enrollment in year 2000 more enrollments N/A 41,516, an increase Qualitative) was 30,000 (PAD) will be recorded of 11,516 over target Date achieved 10/29/2001 11/30/2003 07/27/2004 12/19/2005 Comments (incl. % This achievement must be viewed as a significant increase in project impact. achievement) Indicator target was exceeded by over 300%. Indicator 4 : School councils organized and operational Value 0 greater than 78 N/A 102 organized and quantitative or operational at iii Qualitative) completion Date achieved 10/29/2001 11/30/2003 07/27/2004 12/19/2005 Comments (incl. % Target exceeded by about 30% achievement) Indicator 5 : Number of policy studies completed Value 8 studies plus quantitative or 0 4 N/A additional policy Qualitative) related activities Date achieved 10/29/2000 11/30/2003 07/27/2004 12/19/2005 Comments (incl. % This indicator provides evidence that significant policy development was achievement) conducted through the inputs of this project. See section 3.2. Indicator 6 : Percent primary schools with Portuguese language books Value quantitative or 0 80% N/A 100% Qualitative) Date achieved 10/29/2001 11/30/2003 07/27/2004 04/11/2005 Comments (incl. % Indicator exceeded target. Indicator target was exceeded by 20%. achievement) (b) Intermediate Outcome Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Achieved at approval Revised Completion or documents) Target Values Target Years Indicator 1 : Number of sites secured for project constructed or rehabilitated primary schools Value 99 sites secured, (quantitative 0 Initial target was N/A developed and used or Qualitative) 65 for school construction Date achieved 10/29/2001 11/30/2005 07/27/2004 04/11/2001 Comments (incl. % Indicator target was achieved by 100% achievement) Indicator 2 : Increased NERs Primary NER is Value Primary NER 70% (Bank 80% and the junior (quantitative estimate for 2000). JSE Increased NERs secondary NER, or Qualitative) NER 25% (Bank estimate 30% (derived from for 2000) 2004/05 UNICEF/ MoEC EMIS data) Date achieved 10/29/2001 11/30/2003 04/11/2005 Comments A thorough analysis of MoEC in early 2006 showed that for the 1998/99 school (incl. % year the Primary GER was 89 % and the Net Enrollment ratio (NER) 51. At the achievement) junior secondary level these figures were 44 and 24 percent, respectively. iv G. Ratings of Project Performance in ISRs Actual No. Date ISR Archived DO IP Disbursements (USD millions) 1 03/19/2002 Satisfactory Satisfactory 0.00 2 12/26/2002 Unsatisfactory Unsatisfactory 2.92 3 06/06/2003 Unsatisfactory Unsatisfactory 3.85 4 12/05/2003 Satisfactory Satisfactory 4.54 5 06/24/2004 Satisfactory Satisfactory 9.15 6 12/21/2004 Satisfactory Satisfactory 11.00 7 05/10/2005 Satisfactory Satisfactory 14.85 8 12/19/2005 Satisfactory Satisfactory 18.46 9 12/28/2006 Satisfactory Satisfactory 20.40 H. Restructuring (if any) Not Applicable I. Disbursement Profile v 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Following the post-referendum destruction in 1999, basic services in Timor-Leste were extremely limited and the Government had few resources with which to restore services. Although each of the social sectors was affected, there was an urgent need to reconstruct and revitalize the education system. The most pressing concerns revolved around the need to replace teachers and administrators who had returned to Indonesia, a serious shortage of classrooms, as a large percentage of the stock had been destroyed, and low student achievement. To adequately attend to these concerns, it was necessary to rehabilitate facilities, acquire appropriate textbooks and instructional materials, and to recruit, employ, train and reorganize teachers and administrators. There was also a need to address the fact that public under- investment in education had placed an undue burden of relatively high costs on parents, many of whom were extremely poor. In addition, two post-referendum issues also emerged as concerns: (a) the introduction of Portuguese as a new language of instruction; and (b) a curriculum (Indonesian) that was obsolescent, dense and crowded with low priority learning elements. This project was identified, designed and prepared under difficult conditions in that Timor-Leste was in the midst of establishing and strengthening its government structure. The United Nations Transitional Administration in East Timor (UNTAET) acted on behalf of the country at that time, and was a major stakeholder during the identification, preparation and early implementation of the project. Emergency School Readiness Project (ESRP). The ESRP, the first post-referendum education project in Timor-Leste, was implemented from June 2000 through December 2001. It was financed by the Trust Fund for East Timor (TFET). Ten months into its implementation, 40 percent of damaged classrooms had been refurbished and furnished and 87 percent of the estimated new textbook requirement had been delivered. Although enrollment in the 2000/2001 school year reached pre-1999 totals at the primary level (grades 1 to 6), it failed to do so at both the junior secondary (grades 7 to 9) and in senior secondary levels (grades 10 to 12). Clearly, ESRP, while contributing substantially to sector development, was insufficient to bring about a full renewal of the system. The Fundamental School Quality Project (FSQP). The FSQP was designed as a second-phase project to address the unmet needs of both the primary and junior secondary levels. It was initially funded for the amount of US$13.9 million and later received a supplemental grant of US$6.7 million for a total of US$20.6 million. The Project sought to maintain existing enrollments in primary education while seeking to restore junior secondary enrollments and overall quality of both primary and junior secondary education to pre-1999 levels (see Section 1.2). At the time of FSQP's appraisal, a Timor-Leste Country Assistance Strategy (CAS) had yet to be approved (although CAS discussions had taken place in November 2000). The region did, however, present a MoP to the Executive Directors on a Transitional Support Strategy of the World Bank Group for Timor-Leste (November 3, 2000). That document included three objectives that were supported by the project: (i) in the short-term, focus on the basics, including the provision of health and education; (ii) build local ownership of the development process; and (iii) foster effective donor coordination. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) PDO 1. To support the Government's efforts to maintain the existing level of primary education enrollment and to restore junior secondary enrollments to the pre-1999 levels with the possibility of increase due to return of refugees and growth of school-age population. 1 During project preparation, it was initially estimated that the 1998/99 primary enrollment was about 185,000 students with a 95 percent gross enrollment rate (GER)). Junior secondary enrollment was estimated to be about 30,000 students with a 60 percent GER. Later post-preparation analysis of MoEC data found that for the 1998/99 school year, the primary level GER was actually about 89 percent and the junior secondary level GER was about 44 percent. PDO 2. To continue to recover the quality of primary and junior secondary education by rehabilitating physical facilities and by providing textbooks and instructional materials. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/ justification Although the project received a Supplemental Grant on July 27, 2004, which constituted a formal revision of the project's funding and scope of services, the original PDOs were not changed. Key indicators were necessarily changed to reflect the changes in expected outputs as stated in the Supplementary Grant Agreement (page 3). 1.4 Main Beneficiaries The principal beneficiaries of FSQP were the existing and future primary and junior secondary school-age population. Important stakeholders were teachers, community and parent organizations, and other clients and providers of education. 1.5 Original Components (as approved) Component A ­ Rebuilding school infrastructure, includes: (i) construction of about five escolas basicas (integrated primary and junior secondary schools) at the district level; (ii) upgrading of about nine selected primary and junior secondary schools to the standard of escolas basicas; and (iii) upgrading of about 65 selected primary schools to the fundamental quality standard. The schools will be provided with furniture, water supplies, and toilets. Component B ­ Providing teaching-learning materials, including the acquisition and distribution to primary and junior secondary schools of textbooks, workbooks, and teachers' guides for language learning, learning kits in science and mathematics, and other instructional materials and minor equipment, including typewriters, computers, and peripheral equipment, software and media. Component C ­ Social mobilization and communications, including production and dissemination of promotional materials, delivery of social messages, development and carrying out of training programs, information campaigns, and monitoring, feedback-gathering, and evaluation. Component D ­ Support for policy development, including technical and empirical studies on education issues and policy options, and data management and dissemination of knowledge regarding the development of the education sector. Component E ­ Management and implementation support, including support to management, operational, and administrative activities of the Project Management Unit (PMU). 1.6 Revised Components In early 2004, it was found that some of the original project targets/activities were overly ambitious. The actual construction costs were much higher than originally estimated at appraisal. Consequently, the project team: (a) reviewed plans and construction strategies with the intent of reducing costs; (b) revised targets to complete only priority escolas basicas (integrated primary and junior secondary schools); and (c) reduced the number of primary schools that would be upgraded to the agreed upon standard of 2 fundamental quality. An agreement was also reached with the Grantee to accept a standard school design for use in the project. This design incorporated the use of steel trusses and columns which would extend the life of the buildings and reduce the costs per classroom from US$25,000 to US$15,000, a savings of about 40% from original estimates. With the introduction of the Supplemental Grant Agreement, Project Component A (Schedule 2 of the original Grant Agreement) was formally revised. The revisions are as follows: Part A - Rebuilding of School Physical Infrastructure (1) Construction of about four escolas basicas at selected sites; (2) Sub-component deleted ­ upgrading of nine schools to the standard of escolas basicas; (3) Upgrading of about 35 selected primary schools to the Fundamental Quality Standard; (4) Provision of school furniture, water supply, and toilets to the new and upgraded schools; and (5) Rehabilitation of about 84 rural primary schools. Parts B, C, D, and E - were not changed. 1.7 Other significant changes 1. With the inclusion of the Supplemental Grant, significant funds were added to finance construction. This additional funding enhanced project impact and added to the scope of services provided, particularly in Component A of the project (Section 3.2, PDO1). 2. In May 2005, in an effort to provide more focus on school quality, about US$1.0 million was reallocated from construction to finance additional learning materials. With this change, funding was provided to support the publication and distribution of the CARE International Timor-Leste (CITL) produced Lafeak children's magazine and teacher guides to all primary and junior secondary schools. Lafaek is the magazine in Tinor-Leste that is created specifically for children. Editions for the early grades are printed in the local language, Tetum, and editions for the upper grades are printed in Tetum and Portuguese. The rationale for support for Lafaek came from project experience and M&E findings which showed Lafaek to be the only viable and effective provider of learning materials to rural areas. A recent survey (2006) of local community members and parents, conducted by USAID and the Office of Transition Initiatives (OTI), showed that 75 percent of children and their parents were aware of and understood clearly the subjects that are feature in Lafaek publications. CARE also states that its findings suggest that Lafaek publications have been highly successful in supporting local school programs, improved self confidence and increased interaction between children and their teachers in the classroom. As it relates to improvements in school quality, this activity must be considered one of the project's most important successes (Section 3.2, PDO2). 3. The project was extended three times (Section 2.2). 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Project Identification. Project identification was generally well conceived and implemented. Early project documentation was satisfactory. Project objectives were clearly identified and formulated in a satisfactory manner. The project identification team drew heavily on the immediate-past successes and limitations of ESRP. Future members of the Government, UNTET, the donor community, other stakeholders and civil society were all consulted and their inputs were considered by the project team. Project identification was satisfactory. 3 Project Design. As a second phase to ESRP, the FSQP was designed to address problems that were not adequately addressed in ESRP, namely: junior secondary enrollment, general access to basic education, and the quality of schooling. The project design was clearly based on identified needs. The project also featured significant policy discussion, sector reform and community participation, although a beneficiary survey or stakeholder workshop was not included as a part of the project design. The Project Design was not fully satisfactory. Project documentation has revealed that the project design and preparation teams had a serious concern that the Government's choice of, and strategy for, the introduction of a language of instruction (Portuguese) that may not be appropriate for teaching and learning in rural areas where that language was not widely spoken. However, this concern was not fully discussed or addressed in the project design and the project ultimately provided large quantities of books in that language. At the end of the project this issue remains a Bank concern. The issue probably should have been more fully discussed and if the design team was unsatisfied with the results of the dialogue, a management opinion should have been sought before the investment in Portuguese texts for the upper primary grades was made. Further, the project design was based on an unrealistic two-year implementation cycle. This period seems far too short particularly as Project Component A features a large amount of site acquisition, facilities design, site preparation, construction, rehabilitation and procurement for schools in rural areas. This unrealistic assessment of project completion time must be considered a shortcoming of the project design. This belief is supported by the fact that the project actually took some 57 months to complete. Even after taking into consideration the additional tasks financed by the 2004 project supplement and the delays attributable to civil unrest, the 24 month projection appears to have been unrealistic. Project Preparation. During project preparation, a strong emphasis was placed on monitoring and evaluation (M&E). The project team prepared a clear, well structured set of performance indicators to support M&E. The project team's early and ongoing response to changing implementation requirements should also be positively noted. Project documentation suggests that the preparation teams were well staffed and sufficient time was allowed for the project preparation activity, but the project preparation team underestimated construction costs and contractor capacity, even though the previous project (ESRP) had also experienced the same high costs and contractor problems. It also did not fully address the project's unresolved issues around the language of instruction. These shortcomings forced significant changes in the project during its implementation. Risks and Mitigation. During project preparation, an effort was made to identify risks to effective implementation and to prescribe measures that would be taken to minimize those risks (see PAD, page 20). These risks were generally addressed as part of the M&E activities and were discussed in the ISRs and other documents associated with project implementation. Given the recent experience with ESRP, the PAD should have included additional discussion about the risks associated with: (a) the short implementation cycle planned for the project; (b) shortages of trained labor in rural areas; and (c) the general lack of contractor capacity. These issues were not stated as risks in the PAD and each became key issues during project implementation. Quality at Entry Rating. The project design addressed core issues such as infrastructure, planning and policy, and learning material needs which were appropriate to the circumstances in Timor-Leste. However, given the ICR's concern about the short implementation cycle that was initially planned for the project, the underestimation of the construction costs, the underestimation of contractor capacity, and the project's unresolved issues around the language of instruction, an ICR Quality-at-Entry rating of marginally satisfactory is justified. 4 2.2 Implementation Project implementation was problematic during the early years of the project. Most of these problems were related to shortcomings in management, procurement, construction design and the estimate of construction costs. During the first two years, the project experienced a lack of strong leadership and as a result, the project did not meet project performance indicators and agreed upon schedules. It is also noted that, during this time, many of the national consultants had not been held accountable for timely performance or the quality of their work. It was reported (by the PMU) that during the first two years of project implementation, there existed little staff monitoring or accountability for the quality of services provided by these consultants. This situation led to a crisis in management that was recognized by both the Bank and the Grantee. As a result, measures were taken to change management personnel and replace some team members. As the MoEC became more established and more involved in project implementation, its PMU became more experienced in the implementation process and implementation improved. This was a gradual process and by the second half of the project implementation was quite satisfactory. During the last half of the project the Bank and the PMU developed an excellent collaborative relationship. As a result, project implementation improved to a point where all project objectives and performance indicators detailed in Schedule 5 of the Grant Agreement were met or exceeded. In should be noted that the project was impacted by the civil unrest that started in April 2006 and continued through project closing. The unrest caused delays in construction and was the main reason for the third extension of the project's closing date. Overall, since 2000, the demand for access to schooling has exceeded the capacity of MoEC to provide adequate places in facilities of acceptable quality. While MoEC, through FSQP, has addressed this concern by rehabilitating or building over 100 new schools, there remain several factors that continue to affect the MoEC's efforts to fully address this issue: (i) there are still a large number of displaced persons; (ii) there is substantial population migration; (iii) the fertility rate is 7.8; and (iv) perceptions of the value of schooling are changing and demand is increasing, particularly in the urban areas. During project implementation a number of factors negatively affected the project's school construction program: (i) initial project designs for prototype buildings were found to be unsuitable for the climate and conditions in Timor-Leste; (ii) a number of sites that were initially approved for construction were later discovered to be technically unsuitable ; (iii) the unavailability of clean, safe water supplies was not clear in some initial site assessments; (iv) contractors were sometimes short of funds (due to very limited cash- flows) thus causing intermittent construction delays; (v) local construction procedures/requirements were unclear; (vi) inclement weather and the remoteness of some locations adversely affected initial construction timelines; and (vii) the scale of construction and number of simultaneously operating construction sites sometimes exceeded the capacity of existing qualified contractors ­ resulting, initially, in delays in both procurement of construction services and subsequent construction. When it was realized that some contractors would not (for the foreseeable future) have sufficient financial capital to procure required initial supplies and materials, it was agreed that all construction materials would be procured through separate supply contracts. This decision contributed greatly to the successful implementation of the project's construction activities. By the end of the project, construction costs were significantly reduced from the levels initially experienced by the project. It has already been noted that one of the shortcomings of project design was the underestimation of these early costs. As a result the initial high costs were a surprise to the project team, and these realities forced the project team to initiate a review construction designs and practices. This resulted in an agreement with the Grantee regarding new improved designs and practices which were 5 longer lasting and more cost effective. Construction costs per student were significantly reduced (Table 2.1). Table 2.1 School Construction Costs under the Emergency School Readiness Project and the Fundamental School Quality Project Construction Construction Construction cost Construction cost per primary cost per escola per primary cost per junior school* basica* student place secondary student place ESRP Schools NA NA $750 $1,180 Initial designs $220-250,000 $750,000 $920 $1050 Phase 2 $150-200,000 NA $700 NA Final designs $100-120,000 NA $500-520 $780-800 * Based on 40 students per classroom The Mid-Term Review The Mid-Term Review (January 28, 2003) revealed a number of implementation concerns. Each was addressed effectively. These included: (i) that project implementation was falling behind the original appraisal schedule; (ii) the need for more ministry engagement in the meeting of the Millennium Development Goals, and (iii) the need for more stakeholder/community involvement in important policy initiatives, including curriculum. Adjustments were also made to the school construction program. The project's budget and performance indicators were revised to reflect these changes. Project Extensions The project has had three extensions. The first extension from November 3, 2003, to November 30, 2005, accommodated the addition of supplementary funding and revised project inputs. The second extension, to June 30, 2006, accommodated a revised schedule of school construction and rehabilitation which continued to lag due to (i) delays from labor shortages, (ii) procurement delays; (iii) shipping and transportation delays of construction materials; (iv) social unrest; and (v) problems with quality and timeliness of consultants and contractors. Finally, the closing date was extended a third time to December 31, 2006 to allow for the completion of the construction program which had been further delayed by ongoing civil unrest that continued to plague Timor-Leste for most of 2006. Project implementation is rated as satisfactory. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization The M&E design was generally well conceived and implemented. Outcomes were clearly established and performance indicators were developed. Where feasible, performance indicators were drawn from established data sets that provided for the establishment of baseline comparisons. With these elements in place the M&E design utilized the Bank standard approach to monitor the progress of the project (ISRs, audits, and mid-term reviews). All of these activities were carried out as planned. The documentation reveals that these missions were adequately staffed and the required information was collected, analyzed and presented in the Aide Memoires and ISRs. The information was used to modify the project as needed during project implementation to maximize its impact, and to ensure that it fulfilled its objectives. It should be noted that the modification of the performance outcomes provided for an increase in project rigor and outcomes. M&E is rated as satisfactory. 6 2.4 Safeguard and Fiduciary Compliance There were two safeguard policies identified in the PAD. These included: (a) Environment Assessment (OD .01) and (b) Involuntary Resettlement (OD 4.12). Throughout the project, both were reviewed by the Bank supervision team and did not become an issue. Safeguard Compliance was rated as satisfactory. All fiduciary requirements were complied with during the project. The project was prepared with significant inputs from a financial management specialist. Financial management records were maintained on the project, reviewed during missions, and at project completion. All audits were complete and clear of negative comments or special opinions. The ICR rates Safeguard Compliance as satisfactory. It also rates Fiduciary Compliance as satisfactory. 2.5 Post-completion Operation/Next Phase MoEC established a clear process to manage the transition of a project activity or new/rehabilitated school into its regular operations. First, the activity/school was completed under the project's budget. Second, the completed activity/school was reviewed/inspected by the Ministry, or its designee to ensure that all objectives/requirements are met, including final site inspection for civil works and the final payments for goods or services. Third, any unfinished work or quality concerns noted during the second phase were corrected and reviewed. Fourth and finally, the activity/school was transitioned to regular operations and budgets. Although initially the PMU had the responsibility of managing this transition, it was gradually absorbed by MoEC over the course of the project. There are strong reasons to be optimistic about the potential for sustainability: (i) MoEC is much more firmly established than it was at the beginning of the project; (ii) the project's cost-effective school designs have been fully adopted by MoEC as models for all future school development; (iii) the instructional materials, Lafeak, provided by the project have contributed to a much stronger curriculum model and potential for quality education; (iv) the construction design and quality assurance unit, the Infrastructure Facilities Unit (IFU), is now fully embedded in the MoEC's Directorate of Administration and Finance; and (v) project policy input has already contributed significantly to all major MoEC reforms and to the establishment of the FTI-supported Strategic Plan for Universal Primary Education. It should also be noted that the Government has already committed budgets for the ongoing support of all new and upgraded FSQP schools. Over the course of the next five years the costs of the IFU and of the Lafaek learning materials will be co- financed by the IDA-financed Education Sector Support Project, the FTI-financed Primary Education Support Project and the MoEC. By the end of the five year period these costs will be fully transitioned to the MoEC budget. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation This project supported specific sector-related goals of the UNTAET/Government of Timor-Leste. These goals included improvements of quality and access to primary and junior secondary education. These goals were clearly identified and documented during project identification. Also, as this project was the second phase project to the ESRP, this project was designed to link with the development goals, objectives and some of the activities of that project. At appraisal, a formal Country Assistance Strategy (CAS) for Timor-Leste had yet to be formulated. However, the project did support three of the objectives (total of five), set by the Transitional Support Strategy of the World Bank Group for Timor-Leste (November 3, 2000). The project also sought to reinforce social safety nets through the improvement of access to education services for the poor, and by improving institutional development in basic education, 7 as well as the internal and external efficiency of basic education through wide scale quality improvements in new and existing schools. 3.2 Achievement of Project Development Objectives PDO 1 The first PDO was designed to support UNTAET/Government as it sought to maintain the existing level of primary education enrollment (185,000 students/95 percent GER) and to restore junior secondary enrollment to the pre-1999 level (30,000 students/60 percent GER) with the possibility of increase due to the return of refugees and the growth of school age populations Just prior to the end of the project, available data indicated that primary enrollments have not only been maintained but have actually increased. For the 2004/05school year, the primary GER was estimated to be 119 percent. At the junior secondary level the GER was estimated to be 59 percent (UNICEF/MoEC EMIS school data, 2004/5, 2005 Annual Joint Program Review, MoEC). The ICR rates the achievement of this PDO 1 as satisfactory. PDO 2 The second PDO was established to continue the previous project efforts to recover the quality of primary and junior secondary education by rehabilitating physical facilities and by providing textbooks and instructional materials. Social Mobilization and Communications. Social mobilization teams visited all planned site locations to meet with the communities and to discuss their needs and concerns. During their visits, they also confirmed probably enrollments and existing facilities and teachers, etc. This data provided a basis for final site selection and the obtaining of community inputs to the implementation process. The social mobilization staff also distributed information and publications to the community and assisted in the establishment of parent-community councils to support school maintenance and other school activities. M&E findings suggest that there has been a steady increase in community participation in school affairs including school maintenance and the day-to-day provision of services. This has been primary accomplished through the wide-spread establishment of parent-community councils to support schools. Construction and Rehabilitation of School Facilities. Through new construction and school improvements, the project significantly increased the nation's classroom capacity at both the primary and junior secondary education levels. The project also provided for significant improvements in the classroom learning environment within the project schools. The project supported the: 1. Construction of Four Escola Basicas in the districts of: Ainaro; Ermera; Dili; and Lautem; 2. Construction of 75 Escolas Primárias (EP) ­ primary schools in the districts of: Manatuto; Covalima; Lautem; Liquica; Bobonaro; Baucau; Manufahi; Aileu; Ermera; Oecusse; Ainaro; and Dili; 3. Refurbishment of 23 EPs in the districts of: Manatuto; Liquica; Bobonaro; Manufahi; Oecusse; and Ainaro. 4. Site works at 13 sites for schools to be constructed with Japanese grant aid (12) and Rotary (1); 5. Procurement and distribution of 705 sets of teacher's furniture, each comprising of a desk, a chair and a cupboard; 8,602 size 1 and 7,843 size 2 students' tables; and 17,204 size 1 and 15,686 size 2 students' chairs as furnishings for all completed schools within the project; and 6. Construction of water supply systems for 72 schools with water from existing sources; 43 schools with water from concrete tanks; and 2 schools (in Dili) from deep bore wells. Textbooks and Instructional Materials. Starting in 2004, the project financed a successful effort to provide the Lafaek children's magazine through CITL. FSQP financed six editions a year of each version of: (i) the regular Lafaek magazine - reached 105,000 children in classes 4-6 of primary school and 7-9 in junior 8 secondary; (ii) the junior magazine, Lafaek Ki'ik, - reached 155,000 school children in classes 1-3 and pre-school; and (iii) the teacher's guide - reached 7,500 teachers in primary, junior secondary. FSQP also financed 55,000 sets of Portuguese language texts and teachers guides for grades 4 and 5. This amounts to the full provision of language texts and guides to 100 percent of the grades 4 and 5 classes in Timor- Leste. However, a follow-up survey conducted towards the end of the project revealed that not all of these texts were delivered to schools as intended. Arrangements were made for those that remained undistributed in the offices of district superintendents to be distributed in time for the FY 07-08 school year. Policy Studies. The project financed eight policy papers and a number of other policy-related activities that made a significant contribution to education policy development. Of particular importance was the 2004 National Education Congress which was Timor-Leste's first open forum for the discussion of policy reform in education. The Congress was attended by 600 education stakeholders and practitioners from all parts of the country. The project also supported the preparation of the draft National Education Policy Framework (approved by the Council of Ministers in 2006). The project also financed technical assistance to support the preparation of the Education Law (approved by the Council of Ministers in 2006) and the Organic Law for MoEC (approved by the Council of Ministers in 2006). In addition, policy notes were also prepared on: education finance; strategies for reaching universal primary education; teacher management and performance; learning materials; and institutional capacity development in MoEC. Presentations on the majority of these topics formed the core of the 2006 MoEC Education Week and they contributed to the Strategic Plan for Universal Primary Education by 2015(the SP-UPC). The SP-UPC forms the basis of the FTI-financed Primary Education Support Project (PESP). The project also financed two additional policy papers that focused on policies and procedures related to junior secondary curriculum development. Finally, the project was able to produce a paper that discussed and laid out the Ministry's current policies for integrating teacher training programs and the basis for the teacher competency framework that is being developed under PESP. FSQP also financed a primary school achievement assessment study in 2003. The project also contributed to the institutional strengthening of MoEC by supporting the development and embedding of the Infrastructure Facilities Unit (IFU) in MoEC. This IFU has been responsible for improving the quality, efficiency and cost effectiveness of infrastructure financed by both donors and the Government. The ICR rates the achievement of this PDO 2 as satisfactory. 3.3 Efficiency During project preparation, a cost benefit analysis was prepared for this project. The findings from this analysis may be found in Annex 4 of the PAD. The ICR did not find the initial cost benefit analysis to be convincing as baseline measurement of efficiency for the project and for this reason, there seemed to be little benefit from trying to replicate the study as a post project measurement of efficiency. Of particular concern, it was noted that the data used in the initial study was derived from pre-transition province data collected by Indonesia prior to 1999. The study team actually noted that the data was not believed to be fully indicative of the situation in Timor-Leste when the study was implemented. It was simply the only data set available to the study team. Furthermore, there was no attempt to estimate social rates of return, but instead the study dealt only with the potential economic return to the primary and junior secondary students targeted by the project. 9 Given the lack of data on earnings and productivity in Timor-Leste, it was impossible to conduct a repeat analysis with a reasonable degree of credibility or value added. It is the opinion of the ICR that, at this time: (a) employment and earnings data collection in the remote, rural communities would be difficult, costly to collect and unreliable; and (b) accurate economic development projections, estimates of future labor costs and data identifying priority industries are limited or non-existent. Furthermore, it can be argued that Timor-Leste has/is experiencing (d) periods of social and political unrest; and (e) considerable population movement. All of these factors combine to suggest that such economic analysis of the returns to the project would be difficult to make and if done, would probably be inaccurate and unreliable. However, the team notes that the project did meet its goals for enrollment and quality. Drawing on the international literature, one could reasonably expect that there would be substantial returns in terms of agricultural productivity and other traditional trades, and the basic education provided to students would yield longer run positive externalities in terms of health, fertility, nutrition, and the educational achievement of the next generation. 3.4 Justification of Overall Outcome Rating Rating: Satisfactory All PDOs were achieved. This assessment was based on: (i) strong project performance as measured by the PDO indicators; (ii) evidence of ongoing government commitment and participation during and after the implementation of the project; (iii) effective interagency coordination at all levels; (iv) indications of local community satisfaction with the overall implementation of the project and its outcomes; and (v) indications of continued sustainability for long term project activities. The project met or exceeded its performance indicators despite implementation problems and significant delays throughout the project. There was a periods of time during the project when political unrest caused some construction delays. There were also delays caused by the lack of contractor capacity (physical and financial) to meet the rather stringent implementation schedules required by the project. Somewhat associated with this last issue, there were other delays in procuring construction materials because the contractor did not have the financial capacity to make the orders. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development Recognizing that over time, improved school access and quality have a direct impact on poverty in rural areas, the project has focused much of its resources on the selection of school sites that are located in such areas. The policy reforms also sought to support decentralization of decision-making down to the community level. In addition, the project's support for the children's magazine Lafaek has lent support to children's rights, gender concerns, health and nutrition and family well-being. As the result of its experience with the project, the Government is currently preparing a campaign to encourage more enrollments of girls in both primary and junior secondary education. (b) Institutional Change/Strengthening The Government has passed the MoEC's Organic Law. This law was supported by the project and its policy studies. It has led to the establishment of an Infrastructure Facilities Unit (IFU) within MoEC that is responsible for the planning, scheduling, database maintenance, construction, rehabilitation and ongoing maintenance of ministry facilities. This is a significant change in the capacity of the Ministry that should lead to better coordination of programs to support infrastructure development in the future. (c) Other Unintended Outcomes and Impacts (positive or negative) None noted. 10 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not Applicable. 4. Assessment of Risk to Development Outcome Rating: Negligible The development outcomes of this project are clearly within the mainstream of MoEC and donor priorities. As such, there is little reason to assume that the outcomes would be at risk in the foreseeable future. However, the Government faces continuing challenges with regard to introducing Portuguese as a medium of instruction. In particular, issues related to language skills of teachers, correspondence of materials to a locally relevant curriculum, and nurturing students language ability in the critical first years of schooling pose significant risks. Notwithstanding these concerns, it is the opinion of the ICR that as the Government has placed high priority on the objectives of the project, with continued support and analysis it will work through remaining challenges over time. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Marginally Satisfactory The project was not assessed at entry by QAG. The Bank's performance during project identification, project design and preparation, and negotiation was mixed. The identification mission was able to assist the UNTAET/Government to clearly identify the needs and to produce a concept paper that was designed to effectively meet those needs. The focus and objectives of this paper did not change much during the subsequent stages of project design, preparation and implementation. The Bank's subsequent design and preparation teams underestimated the construction costs for the project, underestimated available contractor capacity, and left unresolved several issues concerning the sequencing and methods for introducing Portuguese as the language of instruction. Adding to that oversight, the project went on to support the procurement and distribution of books and teacher guides in the Portuguese language even though the issue was not resolved to the satisfaction of all parties. Furthermore, the teams supported a two-year implementation period for the project even though the project had plans to implement extensive construction and procurement activities in rural areas. Bank experience has long shown that for such projects, a short implementation period is generally a high risk. This proved to be the case as the project's implementation period was more than twice (57 months) the planned 24 months and the project required three extensions to complete its PDOs. Project negotiations were well planned, managed and implemented. Evidence of this effectiveness may be seen in the Bank teams' ability to work through an unusual grant arrangement whereby the UNTAET served as the grantee and represented the interests of the country and its developing government. (b) Quality of Supervision Rating: Satisfactory The supervision missions, conducted at roughly six-month intervals, with timely and adequately staffed Aide Memoirs (prepared in collaboration with the Government team) were well prepared. It was also found that the supervision teams consistently monitored the safeguards and fiduciary aspects of the project as required by Bank guidelines. The PDO indicators were reviewed and updated during each supervision mission. The project was consciously linked with the previous Bank project and with other education projects with commonly defined objectives. It is noted that the supervision of the project 11 steadily improved through the duration of the project and during the last two years, in particular, became efficient and effective. Throughout the duration of project, the project Task Team Leaders (there were two) effectively utilized regional personnel and consultants. This effectively reduced the cost of project supervision. As noted, procurement of goods and construction was initially a problem for the project. However, due to careful oversight and improved local conditions this improved substantially over the course of the project. During the final months of the project, the FSQP procurement team was recognised publicly as the best in the country and also received an award for this from the World Bank's president. A review of the PSRs/ISRs and associated Aide Memoirs revealed that a number of problems arose during the life of the project that were effectively addressed by the supervision teams through close collaboration with the Government. In all cases, specific actions were agreed upon, with the Government, to bring the project back in line with the implementation plan. Relations with the Borrower have been strongly collaborative and productive. Evidence of these relations is best seen during the past couple of years as the Government has become more fully established and has had more time to focus on project implementation. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory This project has been challenging to the Bank. Aside from the design weaknesses already discussed, the project faced management and implementation problems which were, in part, brought about by the effects of political and financial situation in Timor-Leste. There is evidence that Bank management and project teams made substantial efforts to address the changing needs brought on by these conditions. Project documentation reveals that the project team remained focused on the PDOs and effectively used M&E and the PDO indicators to measure project success as required by Bank guidelines. This was particularly true after the restructuring of the project in 2004. The project team also exhibited a willingness to be flexible, while at the same time maintaining the required rigor of project implementation. The end result was that most issues that arose during implementation were addressed in ways that allowed for the successful completion of the PDOs as measured by the revised PDO indicators. This resulted in raised targets and final outcomes that went beyond those originally anticipated. 5.2 Borrower Performance (a) Government Performance Rating: N/A (b) Implementing Agency or Agencies Performance Rating: N/A (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory Project Identification and Preparation. As a result of the successful impact of the first education project, ESRP, the Bank and the Government agreed to move quickly to prepare and process this project as a second phase effort. This required quick action on the part of both the Government and the Bank to mobilize resources, and to select and field strong teams to identify and prepare the project. Both were successful in meeting this requirement. The Government counterpart team participated actively in the preparation of the project and was particularly effective in the joint consideration of project options and the collection, organization and analysis of the data required to develop the project design. As preparation activities were initiated, the counterpart team made many significant contributions to the 12 quality and scope of the project design. It has been judged that the project was adequately designed and prepared by this combined team. Project Implementation. Initially the project experienced delays due to inadequate management and staff to effectively implement the project. This was resolved during the first quarter of the project. The Government and its provincial and district agencies were also slow to approve and coordinate the necessary activities and local support needed to implement the construction program in the more remote areas. This issue was addressed through a series of Bank/Government meetings and a revised plan was produced that allowed for a more flexible, efficient implementation of the construction program. Once these issues were addressed, a more effective implementation of the project took place. Throughout, the PMU worked closely with the Bank's task team. There is evidence to show that as the project progressed, when problems arose, in procurement for example, the PMU, representing the MoEC, worked closely with the Bank task team to make the necessary adjustments to keep the project as close to schedule as circumstances in Timor-Leste permitted. Procurement was initially an issue, especially related to the procurement of construction materials for new school construction and the rehabilitation of existing schools. Upon review, the PMU found that the cause of the delays could be traced to that fact that many contractors did not have the financial capacity to purchase these goods prior to contract execution. With concurrence from the Government and the Bank, the PMU moved quickly to rework the procurement program to allow for direct contracting for those goods outside of the existing contractors' contracts. This approach effectively addressed the problem and the implementation of the construction program began a positive trend toward the previously set implementation schedule. By the end of the project, the PMU had been able to fully meet and even surpass the planned construction program. This was a difficult management task. The PMU/Government should be commended for its willingness to be flexible, and its ability to effectively address this issue. During Mid-term Review, when there was a need to adjust the implementation plan to better reflect reality, the changes were adequately managed by the PMU, and agreed to by the Bank. As a result, the project was able to increase the overall impact of its development objectives while staying within budget. The PMU also was able to effectively establish computerized project scheduling and indicator tracking systems, M&E systems, and other management procedures that served to enhance the effectiveness of project implementation. In its support of Bank supervision missions, the PMU often prepared operational studies, project performance audits, financial analyses, and architectural plans necessary to support the supervision process. The PMU also consistently provided advanced planning to improve Bank efficiency during its visits to project sites. 6. Lessons Learned (both project-specific and of wide general application) Project Management · Project implementation periods (project duration) should be realistically set to support accurate estimates of the time required to fully implement all the activities planned for the project. If the project's implementation period is set for too short a period, there is a significant risk that the project will have to extended in order to meet its PDOs. Such extensions often require reallocations of funds, impose additional costs on the Bank and the Borrower/Grantee, and impose opportunity costs due to delays in the completion of the project. · When projects fail to make adequate progress, it is important that both the Bank and the Borrower/Grantee quickly understand and agree that the project has a problem. Immediate and effective dialogue between the concerned parties can provide for a quick intervention or adjustment 13 that will bring the project back on track, most usually saving time and resources and lessening the potential of a negative impact; · The introduction of a new language of instruction to a national school system requires careful review and attention before policies are established and implemented. Many factors need to be considered in the context of the county's needs. These include, existing languages spoken by the population, culture, political history, future plans for economic development, and the capacity to provide appropriate pedagogical support and instructional materials. The project offers an example of poor practice in that it supported the provision of Portuguese texts for upper primary well in advance of any associated curriculum reform or curriculum development. Nor did it ensure that these inputs were integrated into a program of instructional materials, teacher training and programmed preparatory language instruction than began in the earlier grades. · Nationally employed consultants should be carefully selected through a vetted process with at least three qualified candidates. The selected candidates should be provided with clear terms of reference and transparent lines of responsibility and reporting. Furthermore, M&E activities should be established whereby the local manager can be informed on a regular basis about the progress and performance of the consultants. · In the education sector, the use of an independent PMU that operate outside the authority of the Ministry of Education is often problematic. Ultimately, the project will need to be mainstreamed into the existing education system and the transition into the system can be difficult if the Ministry of Education does not fully agree with the approach and feel an ownership of the project. It is probably better, where possible, to use a unit (PMU) of the Ministry of Education to implement the project from the beginning. In general, independent PMUs that operate outside of the Ministry are inadequate instruments for achieving capacity building and gaining long lasting results. School Building Designs and Construction · In the interests of efficiency and the establishing of an immediate capacity for a project to have an impact, it is better if facility designs, construction programs and associated procurement concerns can be reviewed, evaluated and agreed in advance of project effectiveness. An absence of such preparation caused substantial delays in early implementation. In situations similar to those that existed in Timor-Leste at the start of FSQP, emphasis should have been placed on building a coordinated team to manage and oversee the construction of appropriate school designs during project preparation. · Where large numbers of similar schools are to be built or rehabilitated by an inexperienced Ministry in association with inexperienced contractors, there is a strong case for the development of a `universal' (but adaptable) building design. The application of this core design will better ensure that quality standards are set, understood, and maintained at a reasonable cost. Books and Instructional Materials · FSQP's association with CITL and the production and delivery of the Lafaek children's magazine confirm that the key elements in effective provision of instructional materials include: (i) solid specifications of requirements, (ii) cost-effective procurement, (iii) safe, secure, well organized inventory/storage, and (iv) efficient, well documented distribution. · The introduction of new language(s) must be carefully planned, monitored and evaluated to ensure that the transition is being effectively introduced and implemented ­ quality of instruction is critical as the timely provision of appropriate learning materials. Teachers must be trained to use new materials and to deliver the new language. 14 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies None (b) Cofinanciers None (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society) None 15 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Components Appraisal Estimate Percentage of (USD millions) Estimate (USD millions) Appraisal /a 1. REBUILDING SCHOOL PHYSICAL INFRASTRUCTURE. 9.80 15.77 161% 2. PROVIDING TEACHING- LEARNING MATERIALS. 1.30 1.24 95% 3. SOCIAL MOBILIZATION AND COMMUNICATIONS. 0.30 0.20 67% 4. SUPPORT FOR POLICY DEVELOPMENT 0.20 0.97 485% 5. MANAGEMENT AND IMPLEMENTATION SUPPORT. 1.00 2.42 242% Total Baseline Cost 12.60 20.60 163% Physical Contingencies 0.70 0.00 0.00 Price Contingencies 0.80 0.00 0.00 Total Project Costs 14.10 20.60 164% Project Preparation Fund 0.00 0.00 .00 Front-end fee IBRD 0.00 0.00 .00 Total Financing Required 14.10 20.60 164% /a The high percentage of appraisal reflects the additional funds (US$6.7 million) provided through the supplemental grants (July 2004). Documentation for these grants showed distribution of the funds by categories, but did not state allocation by components. (b) Financing Appraisal Actual/Latest Source of Funds Type of Percentage of Cofinancing Estimate Estimate (USD millions)(USD millions) Appraisal Borrower 1.5 0.00 .00 Special Financing (IDA Grant) 12.6 20.6 163% 16 Annex 2. Outputs by Component Component 1 (Part A of Grant Agreement) ­ Rebuilding school infrastructure, includes: (i) construction of about five escolas basicas (integrated primary and junior secondary schools) at the district level; (ii) upgrading of about nine selected primary and junior secondary schools to the standard of escolas basicas; and (iii) upgrading of about 65 selected primary schools to the fundamental quality standard. The schools will be provided with furniture, water supplies, and toilets. Outputs 1. Escola Basicas ­ integrated primary and junior secondary schools · Four Escola Basicas were completed. One in each of the following districts: Ainaro; Ermera; Dili; and Lautem. Ten additional schools to be built and upgraded under the original grant agreement were financed from other sources. 2. Escolas Primárias (EP) ­ primary schools · 75 new primary schools were completed in the Districts of Manatuto, Covalima, Lautem, Liquica, Bobonaro, Baucau, Manufahi, Aileu, Ermera, Oecusse, Ainaro and Dili. 15 additional EPs to be constructed under community grants were cancelled under FSQP as they are to be constructed under CFET funding. · 23 EPs have been refurbished in the Districts of Manatuto, Liquica, Bobonaro, Manufahi, Oecusse and Ainaro. 3. Site works · Site works were completed at 13 sites for schools to be constructed with Japanese grant aid (12) and by rotary. 4. Procurement of School Furniture · 705 sets of teacher's furniture, each comprising of a desk, a chair and a cupboard, were provided to all competed schools. · 8,602 size 1 and 7,843 size 2 students' tables were provided to completed schools. · 17,204 size 1 and 15,686 size 2 students' chairs were provided to completed schools. 5. Water Supply · 72 schools were provided with water from existing sources. · 43 schools were provided with concrete water tanks. · Two deep bore wells were drilled for schools in Dili. Component 2 (Part B of Grant Agreement) ­ Providing teaching-learning materials, including the acquisition and distribution to primary and junior secondary schools of textbooks, workbooks, and teachers' guides for language learning, learning kits in science and mathematics, and other instructional materials and minor equipment, including typewriters, computers, and peripheral equipment, software and media. Outputs 1. Lafaek · Starting in 2004, FSQP financed the provision of Lafaek children's magazine. Under contract to MoEC, CARE provided (i) the regular Lafaek magazine to 105,000 children in classes 4-6 of primary school and 7-9 in junior secondary; (ii) Lafaek Ki'ik (junior) to 155,000 school children in classes 1-3 and pre-school; and (iii) a teacher's guide to 7,500 teachers in primary, junior secondary. 2. Textbooks · FSQP financed 55,000 sets of Portuguese language texts and teachers guides for grades 4 & 5. 17 Component 3 (Part C of Grant Agreement) ­ Social mobilization and communications, including production and dissemination of promotional materials, delivery of social messages, development and carrying out of training programs, information campaigns, and monitoring, feedback-gathering, and evaluation. Outputs 1. Site Selection and Community Participation · FSQP-financed social mobilization (SM) staff were assigned to visit all site locations when project engineers assessed possible construction sites. During these visits, the SM staff participated in discussions with the community regarding their needs and concerns about education and the planned school construction/rehabilitation. They also confirmed the probable enrollments and any applicable existing teacher information. In this way, they were able to (i) provide input to the final site selection process, and (ii) obtain community inputs and cooperation relative to the project activities at that location. The SM staff also distributed relevant materials and assisted communities to establish parent-teacher-community councils for school maintenance and other participatory activities. Component 4 (Part D of Grant Agreement) ­ Support for policy development, including technical and empirical studies on education issues and policy options, and data management and dissemination of knowledge regarding the development of the education sector. Outputs 1. Primary School Achievement Assessment Study of 2003 · A sample based achievement testing of students was completed in 2003 as were follow-up interviews with students, teachers, parents, and education administrators. 2. First National Education Consultative Congress · The First National Education Consultative Congress was successfully conducted in October 2004. 3. Policy Studies · By project completion short policy studies were completed in the following areas: (i) curriculum options; (ii) teacher training; (iii) education finance; (iv) education legislation; (v) assessment of capacity to deliver services; (vi) curriculum planning for pre-secondary schools; (vii) options for integration of MoEC's teacher training programs; and (viii) options for the development of learning materials and textbooks. In addition, the project contributed policy input to the drafting of the education law and organic law. Component 5 (Part E of Grant Agreement) ­ Management and implementation support, including support to management, operational, and administrative activities of the Project Management Unit (PMU). Outputs 1. PMU · The project originally financed a PMU which consisted of three international staff including the project director, financial adviser and procurement adviser supported by national staff. In 2004 the MoEC nationalized the project director and finance advisor positions, leaving only the International Procurement Adviser in the PMU. These changes caused some transition problems which were quickly addressed through consultation with the Bank project team. 2. Additional Positions Financed by FSQP · Advisor to the Vice Minister for Primary and Secondary Education · Financial management specialist to assist the Director of Finance and Administration · Advisor to assist with the SP-UPC 18 Annex 3. Economic and Financial Analysis A cost benefit analysis was prepared for this project (Section 3.3). The findings from this analysis may be found in Annex 4 of the PAD. The ICR did not find the initial cost benefit analysis to be convincing as baseline measurement of efficiency for the project for several reasons stated in Section 3.3. The ICR noted that the original cost benefit analysis had used, as its principal source, pre-transition province data collected by Indonesia prior to 1999. This dataset must be considered less than adequate for such an analysis since it almost certainly was not representative of the situation in Timor-Leste at the time of project preparation. When one also realizes that there is little formal employment in the most areas of Timor-Leste, it is hard to understand how one could expect to accurately estimate lifetime earnings (required for estimate of economic RORs) and then attribute those earnings to gains made through primary and junior secondary education when most of the school population is based in rural, sometimes remote areas. For these and the other reasons stated in Section 3.3, no attempt was made to conduct a comparative cost benefit analysis following project completion. Even though the Rate of Return (ROR) is not known and probably cannot be accurately estimated at this time, it is possible to observe that the ROR, whatever it might be, has not been improved by the project. Given the ongoing civil strife, deterioration of the labor market and the economic downturn that has been witnessed in Timor-Leste over the past several years, it is the opinion of the ICR that the ROR almost certainly has decreased. During project preparation, the project did accurately estimate initial school enrollments and identified quality issues associated with primary and junior secondary education. Using these findings as baseline estimates, it is possible to say that both project PDOs were met, based largely on baseline-referenced measurements of quantitative impact. Examples of these baseline figures may be seen in the Project Performance Indicators. It should also be noted that internationally it has long been recognized that when there are significant improvements in basic education, there are many other benefits (externalities) that may be derived from the investment. Examples of these externalities include; (a) long-term improvements in agriculture production; (b) improved fertility rates; (c) improved health and care of young children; (d) improved standards of living in rural areas; (e) higher family incomes and access to discretionary income; and (f) improved community services, particularly in rural areas. 19 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialty Lending (from Task Team in PAD Data Sheet) Alfonso F. de Guzman Senior Education Specialist EASHD Task Team Leader Nurul Alam Senior Procurement Specialist EAPCO Procurement Colin J. Barry Consultant - Teacher Education EASHD Teacher Education Catherine Lisa Campeau Consultant ­ Project EASHD Project Preparation Preparation Filomeno Aleixo da Cruz Project Officer EASHD Project Management Abdul Haji Senior Finance Management EASHD Finance Management Specialist Ronald L. Isaacson Deputy Head of Mission EACDF Governance and Management Parivash Mehrdadi Program Assistant EASHD Headquarters Project Support Karin Nordlander Lead Counsel LEGEA International Law, Gran Agreement, Negotiations Hung Kim Phung Senior Disbursement Officer LOAG1 Grant Disbursement Christopher J. Smith, Jr. Consultant, Procurement and EASHD Procurement Implementation Nigel Wakeham Consultant, School Facilities EASHD Facilities Planning and Design and Construction Construction Kin Bing Wu Senior Education Economist SASHD Education Economics Supervision (from Task Team Members in all archived ISRs) Jerry G. Strudwick Senior Education Specialist EASHD Task Team Leader (since 10/01/2003) Robert Scouller Consultant, Education EASTR Education Technology Technology Stefanie Stallmeister Operations Officer EACDF Operations Management Rui Manuel Hanjam Consultant, Education EASHD Education Technology Technology Bisma Husen Procurement Specialist EAPCO Procurement David Michael Chandler Senior Financial Management EAPCO Financial Management Specialist Ian Collingwood Curriculum Specialist EASHD Curriculum 20 Audrey Aarons Teacher Development/ AFTH3 Teacher Development/ Curriculum Curriculum Development Development Specialist Ronald L. Isaacson Deputy Head of Mission EACDF Governance and Management Alfonso F. de Guzman Senior Education Specialist EASHD Task Team Leader (prior to 10/01/03) Kye Woo Lee Consultant Economist EASHD Education Economics Emmanuel Y. Jimenez HD Sector Director EASHD Management (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle No. of staff weeks USD Thousands (including travel and consultant costs) Lending FY01 18 129.15 FY02 5 17.77 FY03 0.00 FY04 0.00 FY05 0.00 FY06 0.00 FY07 0.00 Total: 23 146.92 Supervision/ICR FY01 0.00 FY02 1 8.93 FY03 8 74.86 FY04 8 53.67 FY05 13 68.50 FY06 19 78.19 FY07 2 15.56 Total: 51 299.71 21 Annex 5. Beneficiary Survey Results Not Applicable 22 Annex 6. Stakeholder Workshop Report and Results Not Applicable 23 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR The following comments have been inserted without modification or editing as per World Bank guidelines. 1. INTRODUCTION 1.1. After the referendum in August 1999, as part of the program of rehabilitation and reconstruction for Timor-Leste a program was developed for the education sector. The first education project financed by TFET, the Emergency School Readiness Project, ESRP (June 2000­December 2001), rehabilitated school buildings to a basic operational level. The Fundamental School Quality Project (FSQP) became effective on May 1, 2002, with a Grant of US$13.9 million and was scheduled to close on November 30, 2003. On November 28, 2003 the closing date was extended to November 30, 2005. A Supplemental Grant valued at US$6.7 million was approved on September 23, 2005 and the closing date was extended to June 30, 2006. Due to the unrest in the Country during April/May 2006 approval was given on June 14, 2006 to extend the closing date to December 31, 2006 to allow the completion of the construction program. 1.2. The project has two objectives: (i) improving access ­ by maintaining the existing level of primary education enrolment and by restoring junior secondary enrolment to pre-1999 levels, through, among others, the rehabilitation of physical school infrastructure; and (ii) improving quality of primary and junior secondary education. In addition to developing the stock of school buildings, FSQP supports the provision of learning materials, social mobilization and communications, curriculum development, policy studies, and project management. 1.3. Cooperation between the World Bank and the MoEC, in particular, is not new. The cooperation between the two parties dated back as far as June 2000 with the First Education Loan amounting to USD13.9 million for the Emergency School Readiness Project (ESRP). The FSQP, grant package amounting to USD20.6 million is the largest thus far allocated to this sector. 1.4 The following is the summary of the objective, scope and achievement of the FSQP. The detailed description of the Project and its achievements by components are contained in the attachments provided. 2. ACHIEVEMENT OF PROJECT OBJECTIVE 2.1. FSQP has satisfied each of the project performance indicators detailed in Schedule 5 of the Grant Agreement and each of the new targets (informally) set at the Mid-Term Project Review as well as the outputs detailed under the September, 2005 agreement to extend the project closing date and reallocate grant funds among project categories. 2.2 To achieve these two objectives, the Project focused on: i) undertaking civil works projects built to a fundamental quality for primary schools in different parts of the country, (ii) acquisition and distribution to primary and junior secondary schools of textbooks and workbooks for the teaching of Portuguese as a foreign language and other instructional materials such as the Lafaek children's magazine, (iii) support for policy development in the areas of curriculum development, teacher training needs study, school network action plan with a partial updating of the school map (when project schools are surveyed) and monitoring and evaluation system for the development of the education sector. In support of Policy Development and the SP-UPC was the completion of two additional advisory notes: options for the integration MOEC's teacher training programs, and options for the development of learning materials and textbooks. 24 2.3. All of the activities undertaken by the project subcomponents had met the target and in some cases had exceeded the initial plan. Modifications to designs for the schools were carried out in the initial stages to achieve economies of scale and the separation of construction and the supply of the major material components achieved a marked reduction in the construction program. 3. PROJECT SCOPE AND ACHIEVEMENT 3.1. The Fundamental School Quality Project consists of 5 main components, namely; A) Rebuilding School Physical Infrastructure, B) Provision of Teacher Learning Materials, C) Social Mobilisation and Communications, D) Support for Policy Development and E) Management and Implementation Support. Part A: Rebuilding School Physical Infrastructure 1. Construction of about five Escola Basicas at selected sites. 2. Upgrading to the standard of Escola Basica of about nine selected schools. 3. Upgrading of about 65 selected primary schools to the Fundamental Quality standard. 4. Provision of school furniture, water supply and toilets to new and upgraded schools. Following the Mid Term Review carried out in February 2003 there were changes to this Component as follows: 1. Of the 14 escola basicas of the project (5 new, 9 up grade), 4 new schools would be constructed following architectural plans being revised to achieve economy and financed from the Grant; the other 10 schools will be financed from other sources. 2. 65 primary schools, with plans revised for economy, would be constructed or upgraded; the upgrading of existing buildings would be preferred over construction of new buildings; 13 more primary schools would be constructed or upgraded in poor rural areas at the rate of one school per district. 3. Provision of school furniture, water supply and toilets to new and upgraded schools. Following ongoing reviews the current position as at 30 November, 2006 is as follows: 1. Construction of four Escola Basicas in Ainaro (Ainaro District), one in Gleno (Ermera District), one in Hera, Dili District and one in Los Palos (Lautem District). Each of these schools have been completed and handed over to the communities. 2. Construction of Primary Schools / Escolas Primárias (EP) 2.1 New construction · 75 new EPs have been completed and handed over to the communities in the Districts of Manatuto, Covalima, Lautem, Liquica, Bobonaro, Baucau, Manufahi, Aileu, Ermera, Oecusse, Ainaro and Dili. 25 · 15 schools to be constructed under community grants have been cancelled under FSQP and will be constructed under CFET funding. 2.2 Refurbishment · 23 refurbished EPs have been completed and handed over to the communities in the Districts of Manatuto, Liquica, Bobonaro, Manufahi, Oecusse and Ainaro. 2.3 Site works · Contracts placed and completed for 12 sites for schools to be constructed by Japanese grant aid. 3. Procurement of Construction Materials One of the major delays in the earliest stages of construction was recognized that the construction companies did not have sufficient cash flow capabilities to procure the construction materials in line with the agreed construction program. Accordingly in 2004 a decision was made to extract the major components from the construction contracts and call for bids for the supply only as detailed below: · Contract for window frames, louver carriers and glass blades placed on 14 October, 2004 with delivery to Contractors as and when required. · Contract for wooden doors and door frames placed on 1 October, 2004 with delivery to Contractors as and when required. · Contract for door and plumbing hardware placed on 1 October, 2004 with delivery to Contractors as and when required. · Contract for Roofing sheets and sisalation placed on 11 November, 2004 with delivery to Contractors as and when required. · Contract for steel roof trusses placed on 15 November, 04, with delivery to Contractors as and when required. 4. Procurement of School Furniture · Contract for 8,602 size 1 and 7,843 size 2 students' tables placed on 14 October, 2004 with delivery finalized for all completed schools. · Contract for 17,204 size 1 and 15,686 size 2 students' chairs placed on 14 October, 2004 with delivery finalized for all completed schools. · Contract for 705 sets of teacher's furniture, each set comprised of 1 desk + 1 chair + 1 cupboard, placed on 14 October, 2004 finalized for all completed schools. 5. Water Supply The project has resources to finance 72 school water systems with an additional 14 schools planned to be supplied by deep bores. This work was to be carried out under FSQP, however following the completion of two deep bores within the Dili District a meeting was arranged between the Ministry of Natural Resources and the Project and it was agreed that the supply of water to schools was the responsibility of the Ministry of Natural Resources and the Project would connect the supply to the infrastructure constructed under the Project. 26 FSQP then selected the priority schools that required ground water tanks and contracts were let to construct concrete tanks to 43 sites. 6. The Organic Law for MoEC will lead to the establishment of an Infrastructure Facilities Unit (IFU) responsible for the planning, scheduling, database maintenance, construction, rehabilitation and ongoing maintenance of ministry facilities. The IFU has been requested to assist the Public Works Department to undertake the design, contract documentation and supervision for schools to be constructed under CFET budget. As no funds were available for FSQP to undertake this work, it has been agreed that the IFU be financed from funds originally allocated for the construction of 15 FSQP schools to be constructed using community grants. Part B: Provision of Teacher Learning Materials Acquisition and distribution to primary and junior secondary schools of textbooks and workbooks for the teaching of Portuguese as a foreign language and other instructional materials such as the Lafaek childrens magazine. Lafaek Starting in 2004, the project has financed the provision of Lafaek children's magazine. Under contract to FSQP/MoEC, CARE will provide: (i) the regular Lafaek magazine to 105,000 children in classes 4-6 of primary school and 7-9 in junior secondary; (ii) Lafaek Ki'ik (junior) to 155,000 school children in classes 1-3 and pre-school; and (iii) a teachers guide to 7,500 teachers in primary, junior secondary. FSQP's support for this program has been critically important as the magazine remains the only teaching and learning material to reach all classrooms. However, FSQP's contract with CARE concluded in November 2005 and the Government has been urged to find Government funds for Lafaek and/or a more permanent source of donor financing. TEXTBOOKS In 2003 and 2004 FSQP provided 55,000 sets of Portuguese language texts and teachers guides for grades 4 & 5. In late 2004 FSQP undertook a review of the use by teachers of these books that had been distributed. The books were designed for learning Portuguese as a second language and were, in some editions, adapted to suit Timor-Leste culture. The evaluation revealed uneven and sometimes inadequate distribution of the materials. Teachers received no training for the approaches used in the books and a problem was noted in respect to the lack of progression between the different textbooks provided for the two years (the lower books used an oral dialogue methodology, while the higher books used a more traditional grammatical approach). A further study needs to be conducted by the Ministry before any further textbooks are introduced. Each of the 4 escola basica built by the project were to receive simple science equipment from the project. These were to be designed to facilitate classroom instruction in line with the science training provided to teachers by other donors. However, this component was deleted from the project. Part C: Social Mobilization and Communications 27 Production of communications materials, carrying out of communications activities, including information campaigns and meetings, and carrying out of evaluation activities, including provision of technical assistance. At the time of the Mid Term Review no substantial activity has been carried out in this component. It was agreed that the social mobilization staff of the project would go to the field when project engineers scope out works, to conduct consultative processes with the community, confirm enrollments and teacher information, help select project sites, obtain cooperation in school construction/rehabilitation and help organize parent-teacher-community councils for school maintenance and other participatory activities. The current situation is that the MoEC did not wish to continue with the FSQP funded international consultant after 30 November, 2005 and has assigned two staff to work with the two national FSQP consultants. However, we have been advised that UNICEF is considering the funding of an international position. Priorities will be to establish the Information & Communications Unit (ICU) and establish unit procedures and mechanisms. Immediate tasks are to include (i) an evaluation of the recent UNICEF financed enrollment campaign; (ii) the preparation of a long-term national campaign to boost and maintain primary enrollment (importantly, girls' participation); and (iii) the development of an effective Ministry wide internal communications. Part D: Support for Policy Development Carrying out of workshops, study tours and studies related to policy development in education, including provision of technical assistance. Following the mid term review it was decided to include a new subcomponent for Curriculum development. This new subcomponent would assist MEC in developing a national curriculum development mechanism and framework that ensures Timor-Leste ownership at all levels and adequate provision of high quality and appropriate teaching and learning materials. The following is a summary of the status of project-supported studies: 1) Primary School Achievement Assessment Study of 2003 (No. 1)-The achievement testing of children and follow-up interviews with students, teachers, parents, and education administrators have been completed and entered in MEC computers. 2) First National Education Consultative Congress was conducted in July-August 2003. 3) During the midterm review, it was agreed that the following additional policy study topics should be included: (i) teacher training needs study; (ii) school network action plan with a partial updating of the school map (when project schools are surveyed); and (iii) monitoring and evaluation system for the development of the education sector. 4) At present papers have been finalized for the following studies: · Short Term Curriculum Specialist; · Curriculum Planning Adviser for Pre Secondary School. · Costing Policy; · Education Finance Specialist; · Legislative Specialist; · Accreditation Specialist; 28 · Teacher Training Mathematics; · Teacher Professional Development; · Assessment of Capacity to Deliver Services; · UPC. Next steps for FSQP in support of Policy Development and the SP-UPC was the completion of two additional advisory notes: (i) options for integration MOEC's teacher training programs, and (ii) options for the development of learning materials and textbooks. Part E: Management and Implementation Support Strengthening the Project Management Unit, among others in financial management, education coordination and Project management, through provision of technical assistance and training. The Unit originally consisted of International staff as Project Director, Financial Adviser and Procurement Adviser supported by National Staff. In 2004 the position of Project Director was transferred to the Minister, the Financial Adviser position was taken over by a National Finance Specialist and the International Procurement Adviser remained as Procurement/Project Management Adviser. FSQP also finances an advisor to the Vice Minister for Primary and Secondary Education and a financial management specialist to assist the Director of Finance and Administration. Furthermore, an advisor was appointed to assist with the SP-UPC. FSQP has satisfied each of the project performance indicators detailed in Schedule 5 of the Grant Agreement and each of the new targets (informally) set at the Mid-Term project Review as well as being on track to complete the outputs detailed under the September 2005 agreement to extend the project closing date and reallocate funds among project categories. 4. MAJOR FACTORS AFFECTING IMPLEMENTATION AND OUTCOME 4.1. Generally most of the activities planned under the Project were successfully implemented and some components had even exceeded their target. However, there were some shortfalls and untimely completion of the projects. Following are among the main factors affecting the implementation. 4.2. Civil Works a. Insufficient technical site investigation hampered implementation process in which some of the sites were later classified as problematic. b. Requirements by some local authorities were not fully understood by implementers causing delays during site possession. c. Initial designs of the prototype buildings did not necessarily reflect the conditions in Timor-Leste and needed a number of revisions to cut costs without reducing quality. d. Construction delays due to poor cash flow management by the contractors. e. Some sites were remotely located and were constantly affected by logistic difficulties, bad weather and poor accessibility. f. Selection of school sites without sufficient review in the accessibility to water supply. g. Insufficient number of qualified Contractors to carry out the vast number of schools tendered during a short period resulted in some schools being rebid. h. During the period April-June 2006 there was civil unrest in Timor-Leste which resulted in the construction of the final 12 schools being delayed. To allow the completion of these schools the closing date for FSQP was extended from 30 June 2006 to 31 December 2006. 29 4.3. Goods As mentioned earlier one of the major delays in the earliest stages of construction was recognized that the construction companies did not have sufficient cash flow capabilities to procure the construction materials in line with the agreed construction program. Accordingly in 2004 a decision was made to extract the major components from the construction contracts and call for bids for supply only. This action minimized delays dramatically and the only problem was the supply of plumbing and door hardware out of Australia which resulted in some delays and the implementation of liquidated damages against this company. The steel roof trusses, roof sheets, doors and window frames were all manufactured locally to a very high quality and delivered on time which helped the construction program to remain on target. The supply of school furniture was delivered on time and warehoused and delivered to schools upon their completion. The only problem encountered was with the school desks which were not as stable as required so a solution was agreed with the supplier who rectified all of the problems. 5. SUSTAINABILITY 5.1 Following the success of FSQP, particularly in the area of construction/rehabilitation of school infrastructure, the sustainability of the project will continue into the Strategic Plan for Universal Primary Completion (SP-UPC) by 2015. With the establishment of the IFU this will enable the planning and construction of the required facilities to enable the availability of class rooms to satisfy the SPU-PC. 5.2 The numerous benefits derived from the Project are continued into the PESP and ESSP and funds would be made available to enhance and upgrade them in future five-year national development plans. The sustainability of these projects is essential to enable the MoEC to continue to contribute to the development of the required pool of skilled and knowledgeable human resources, learning materials and construction of required facilities and to enable the fulfillment of the national aspirations towards SP-UPC. 6. BANK AND BORROWER PERFORMANCE 6.1 The Ministry of Education & Culture wishes to express appreciation to members of both the World Bank and the MoEC for having successfully implemented the FSQP. The success is attributed to the close cooperation forged between the Bank team and MoEC personnel entrusted with the FSQP as well as the manifestation of the series of thorough and effective supervision on the Bank part, the commitment of MoEC personnel towards the project and close monitoring and leadership of the Project Management Unit (PMU). 30 BANK PERFORMANCE 6.2 The Bank's supervision mission team under the leadership of the task manager had provided constant encouragement and motivation to the MoEC officials involved in the FSQP. The supervision mission had also been very responsive and supportive in resolving implementation issues that required Bank approval or NOL. Their continuous support and ready assistance as well as their helpful comments and feedback had also assisted the implementers to be consistent in their project implementation, especially with respect to monitoring and evaluation, which are important aspects of the project cycle. 6.3. The team had also helped improved implementers understanding of the Bank's guidelines and procedures in operating the projects. BORROWER PERFORMANCE 6.4 All those involved in the Project had continually strived to implement the project in the best possible way. Nevertheless, there were some setbacks that had affected the performance of the project in the early stages such as project management and design problems. However, the outcome of the project is overwhelming, so much so that the implementation problems faced earlier seem negligible. All the implementers had performed well despite the early problems at beginning of project implementation. Their commitment and the assistance from the Bank team had enabled them to abide to the World Bank's procedures and guidelines and had succeeded in completing the project within the loan extension period. 6.5 The appointment of a project manager, architects, technical assistance in construction design and monitoring, a quantity surveyor and procurement staff to the Construction Management Unit (CMU) contributed to the success of the tremendously huge civil works project. The outcome of this performance was recognized by the World Bank and the President of the Bank presented an award to the project for excellence in procurement. 7. LESSONS LEARNED At the beginning the project suffered from a lack of strong leadership and as a result, the project continually failed to attain key goal indicators, such as task schedules and cost effective designs. More importantly though, the lack of strong leadership and management systems ultimately led the project to a crisis point where drastic measures were necessary to improve the project. These measures were only possible after there was a change in management personnel and the replacement team was able to recognise the inefficiencies. It is my opinion that the issues, which affected the projects performance, can be summarised, by saying that a large percentage of the national consultants had previously not been required to commit themselves to targets or objectives, during both the Indonesian occupation and the original FSQP management team. During the first 2 years of the project there was very little staff monitoring or accountability of poor work ethics or performance. Subsequently staff felt they were under no obligation to improve or change and this `attitude' made it extremely difficult for the new management to maintain an acceptable level of success. 31 Project Appraisal Document ( PAD ) The PAD is intended as a form of "client brief" from the World Bank to the consultants involved with design of the building structures, as such it would be beneficial to have a detailed document that provided the consultants with a framework within which to base the building designs. For example, if the World Bank had previous school building designs from similar projects, which had been cost effective and pragmatic to build, then perhaps these designs could have been included as a guide to the "quality" of expected designs. Similarly if the PAD had outlined the areas of school design, which had priority, this would have had the effect of focusing the design team ­ i.e., if the PAD had mentioned that the priority of the design was: · to provide a structurally durable building with a consideration to aesthetics. · external space / amenity is secondary to durability · Material usage/detailing to be kept to the minimum possible for durability / functionality. FSQP endured many difficult design iterations in the early years, due to the Management and Architectural teams interpretation of the PAD ­ the early Architectural designs were extravagant and impractical with respect to the environment in which the buildings were to function. It is my belief that if the PAD had been more rigid in its description of the "scope of works" then the design team would have been more controlled. Furthermore, the original architects adopted a work ethic where the team would arrive at work between 9 ­ 10am leave at 12 noon for 2hrs lunch, have coffee 6 ­ 7pm and then work until 10pm. This unfortunately did not correspond to the other design internationals who attended work from 8am ­ 12noon then 1pm ­ 7pm. This uncoordinated team approach did at times delay design decisions and did not allow effective management of the National design team. After two years of the Project the original Project Director left and the position was transferred to the Minister. This was not effective as the Minister did not have the time available to have input into the project and decision making was severely hampered. The areas where there was very little impact were Social Mobilisation and supply of textbooks. The Social Mobilisation Unit had a number of International Advisers over the term of the Project, however the Ministry did not provide any productive direction or input to the extent that the Project ceased funding this activity. In 2003 and 2004 FSQP provided 55,000 sets of Portuguese language texts and teachers guides for grades 4 & 5. In late 2004 FSQP undertook a review of the use by teachers of these books that had been distributed. The books were designed for learning Portuguese as a second language and were, in some editions, adapted to suit Timor-Leste culture. The evaluation revealed uneven and sometimes inadequate distribution of the materials. Teachers received no training for the approaches used in the books and a problem was noted in respect to the lack of progression between the different textbooks provided for the two years (the lower books used an oral dialogue methodology, while the higher books used a more traditional grammatical approach). A further study needs to be conducted by the Ministry before any further textbooks are introduced. 32 The main areas of success for the project was the Lafaek magazine and the establishment of a professional Infrastructure Facilities Unit. Starting in 2004, the project has financed the provision of Lafaek children's magazine. Under contract to FSQP/MoEC, CARE will provide: (i) the regular Lafaek magazine to 105,000 children in classes 4-6 of primary school and 7-9 in junior secondary; (ii) Lafaek Ki'ik (junior) to 155,000 school children in classes 1-3 and pre-school; and (iii) a teachers guide to 7,500 teachers in primary, junior secondary. FSQP's support for this program has been critically important as the magazine remains the only teaching and learning material to reach all classrooms. Under FSQP there has been the establishment of an Infrastructure Facilities Unit (IFU) responsible for the planning, scheduling, database maintenance, construction, rehabilitation and ongoing maintenance of all ministry facilities. Over the past two years this unit, under the management of the current Project Manager, has been transformed into a very professional unit which enabled the project to meet all of the deadlines in relation to the civil works component of the Project. Furthermore, this unit has designed a "prototype" building design which is of a very high standard at a relatively low cost. 33 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Project's Procurement Team Recognition ­ Comment by Steffi Stallmeister, EACDF The Infrastructure Facilities Unit (IFU) in the MoEC has played an important role in rebuilding the education system. It has been recognized for maintaining a high level of transparency in its procurement procedures. The procurement process under the TFET financed Emergency School Readiness Project (ESRP) and FSQP has continuously been improved. The most critical steps include: (i) attempts of corruption and collusion have been openly discussed in order to create awareness of this practice, (ii) close control over cost estimates, and (iii) alleged collusion attempts have been submitted for further review and investigation to INT. Cases have also been referred to appropriate local authorities such as the Inspector General's Office and the Procurement Office. Moreover, once awarded, contracts are supervised meticulously, with schools under construction visited several times a week by construction supervisors. The IFU has also been very proactive in educating contractors in preparing and submitting bids by systematically holding pre-bid meetings, ensuring a smooth procurement process. The IFU is a model program with (i) a clear, single focus, (ii) highly qualified technical assistance and staff, (iii) accountability and transparency, and most importantly, (iv) government ownership and commitment. Project Impact ­ Comment by Ginny Kintz, CARE International Timor-Leste CARE International Timor-Leste supports the education sector through the production and distribution of supplementary curriculum materials for all pre-primary, primary and pre-secondary schools in Timor- Leste. Since 2004, the funding for these materials has been provided through FSQP. Because its work is national in scope, CARE is well placed to observe the impact of donor initiatives, including FSQP. The most visible impact, of course, is the reconstruction and rehabilitation of schools. CARE conducts regular monitoring of the distribution and use of its learning materials in all districts. It was notable that we saw school construction being carried out under FSQP across the country, in remote areas as well as in communities that were more easily accessible. It is clear that there was a consistent standard of quality in all schools built or renovated. The provision of school furniture is another very tangible result. We observed significant differences in the teaching and learning atmosphere in schools where previously children were sitting on the floor or on stools that they carried from home every day. In the production of learning materials, CARE works very closely with the Vice Minister and other staff of the Ministry of Education. Thus we are frequently at the MEC offices and have the opportunity to observe change over time. The SP-UPC, which was developed with support from FSQP, now provides a consistent focus both for MEC and its partners. Consultants supported by FSQP provided insight and direction that benefited all of us working within the education sector. The concrete, long-term support provided by FSQP both in policy development and in management and program implementation has contributed significantly to an increased level of professionalism within MEC. 34 General Feedback on Project implementation ­ Comments by Baba Danbappa, Deputy Representative, UNICEF Dili ­ Timor-Leste Component 1 - Construction: Reconstruction and rehabilitation of schools and classrooms has been a major challenge and the FSQP project has contributed towards ensuring that safe and sufficient learning spaces are provided to school going children. The establishment of an infrastructure unit to plan and implement school construction will strengthen the ministry's capacity to provide additional classrooms and new schools where required. Challenges that remain to be addressed: · The lack of a functioning school maintenance system to channel resources to schools and PTAs for basic repairs results in minor damages becoming major damages as communities does not have the resources to manage such repairs. · The reliance on communities and local administration to provide water for the toilets built under the FSQP has not been viable. Many schools cannot open toilets to children due to the lack of water. · Timor-Leste's geographical condition varies from hot humid conditions in coastal areas to cold dry or wet conditions in the mountains. Thus building the same school structure in all regions may not adequately address the climatic and geographic needs. Component 2 ­ Learning Materials: Lafaek magazine has been and continues to be the most read and used resource in schools in Timor-Leste. The magazine provides both entertainment and education to children and is very popular. Through regular evaluations carried out on behalf of the World Bank, there has also been a continuous process of reviews and revisions based on evaluations and lessons learned. This has contributed to relevance and importance of Lafaek as a learning resource. UNICEF's Child Friendly Schools Project's promotion of peer learning structures at school and at school cluster level has been working closely with Lafaek for Teachers to encourage teachers to try new techniques or activities such as those listed in the Lafaek for Teachers. The provision of textbooks in Portuguese to schools has been well documented in a World Bank funded evaluation in 2004. The increasing level of Portuguese language skills amongst teachers is contributing towards increased use of Portuguese language textbooks by teachers in schools. Challenges that remain to be addressed: · The levels of skills amongst teachers on working with textbooks in a child centered and participatory manner are low. This is a challenge which UNICEF is addressing through the school level teacher education and peer learning initiatives under the Child Friendly Schools project. · Although nearly all teachers have or are attending Portuguese language classes and have increased levels of Portuguese, these teachers have not been equipped to teach Portuguese to children who do not have any knowledge of this language. This is a challenge facing all partners working with the Ministry of Education and Culture. · The development of structures, systems and capacities for the procurement of textbooks, design of teaching learning materials and textbooks as well as the distribution of these materials to the schools remains a critical bottleneck in the ministry. Component 3 - Social Mobilization: 35 Very little information has been available about this component to other donors, so UNICEF is not in a position to comment on this component. UNICEF has worked with the social mobilization unit in 2005 to organize the "back to school" campaign and to organise workshops for journalists on education and the right to education for all children. Component 4 - Support for policy development: UNICEF worked closely with the World Bank in organizing the 2004 Education Conference and in the drafting of the SP/UPC. There has been close and collaborative partner interaction which has resulted in all partners recognizing the SP/UPC document as the framework for development cooperation since 2006. The studies prepared with World Bank support have been valuable sources of information to guide discussions for policy and planning. Challenges that remain to be addressed: · Given the relatively low absorption capacity of key staff in the ministry, greater effort is required to ensure that studies are reviewed in a joint process with these staff. This was done with the SP/UPC document and has resulted in better understanding and ownership than with any previous documents. Component 5 - Management and program implementation support The provision of advisors to the Vice-Minister of Education (School Education) has contributed towards strengthen coordination with donors and has also contributed towards the establishment of the weekly coordination meetings of all divisional directors which was introduced in the ministry in 2006. There has also been increased preparation through joint review of documents which has contributed to more informed decision making processes. The joint drafting of terms of reference and joint selection of the teacher training advisor has been a very positive collaboration which is ongoing as UNICEF provides financial support to host workshops on the competency framework as well as to follow up on activities agreed upon during the absence of the advisor. This has resulted in continued work on the competency framework despite the political unrest which resulted in cancelled and delayed visits by the advisor who continues to be in regular contact with UNICEF. Very little information is available about the work of the advisors on financial management. This area is a real challenge which UNICEF also needs to address as current processes of providing resources directly to programme divisions for activity implementation are not viable. Challenges that remain to be addressed: · Coordination and information sharing about the technical assistance to be provided, job descriptions and arrival of advisors remains a challenge which hampers donor coordination. · Joint programming and pooling of resources can be further strengthened to ensure that technical support is provided in a timely and sustained manner. 36 Annex 9. List of Supporting Documents Project Concept Note Project Appraisal Document, September 28, 2001 Trust Fund for East Timor Grant Agreement - Fundamental School Quality Project, October 29, 2001 Supplemental Grant For the Fundamental School Quality Project, July 27, 2004 Memorandum of the President Minutes of Negotiations Project Status Reports (PSR) Mid-term Review Report and Agreements, March 4, 2003 Implementation Status Reports (ISR) Mission Aide Memoires Back-to-Office Reports (BTOR) Project Financial Reports Project Correspondence Background Paper for Donor's Meeting on East Timor, Brussels, December 5-6, 2000 37 124° 125° 126° 127° Wetar Island I N D O N E S I A 10° 10° Romang Island TIMOR- LESTE Atauro Island Biquele t Alor t r a i Atauro S Island Berau W e t a r Cape Tei Lautém Com Tutuala Rantar Laivai Maínal Maínal Cape Cutchá Jaco Island Baucau Bucóli Fuiloro Manatuto Island Los Palos DILI Metinaro SeicalBAUCAULaga Vermasse er Liquiça DILI Lacló Lacló Laleia Luro LAUTEM Maubara Cape Bazartete Venilale Quelicai Corimbala LIQUIÇA LIQUIÇA Baquia Railaco AILEU Laclo MANATUTO Uato Carabau Loré Ermera Cape de Loré Ossu Iliomar Fatu Beso Laclubar Savu Sea Aileu Lacluta Uatolari Atabae ERMERALetetoho VIQUEQUE Turiscai Aliambata Maubisse Viqueque BOBONARO Tata Mai Lau (2963 m) Hato Builico Sahen Barique Batugade Bé Aco Maliana MANUFAHI CFato Berliu Cape Deilubún 9° Ainaro Bobonaro Same iere Umaboco 9° Alas a n d AINARO Mape Sacato Hatudo Oecussi I s l Loloto Fatolulic Betano Cape COVA LIMA Calétec OECUSSI r Fohorem Kitrana Nítibe Nítibe Oé Silo Oé o Suai m Tilomar 127° Passabe i Cape Tafara T To Naikliu Timor Sea TIMOR-LESTE I N D O N E S I A SELECTED CITIES AND TOWNS DISTRICT CAPITALS 0 10 20 30 Kilometers NATIONAL CAPITAL 0 10 20 30 Miles RIVERS 8° JANUAR MAIN ROADS IBRD To This map was produced by the Map Design Unit of The World Bank. Kuala The boundaries, colors, denominations and any other information DISTRICT BOUNDARIES Lumpur shown on this map do not imply, on the part of The World Bank 33496 Y 2005 Group, any judgment on the legal status of any territory, or any INTERNATIONAL BOUNDARIES endorsement or acceptance of such boundaries. 124° 125° 126°