Document of The World Bank Report No: 29152 IMPLEMENTATION COMPLETION REPORT (SCL-41600) ON A LOAN IN THE AMOUNT OF US$143.4 MILLION TO THAILAND FOR A UNIVERSITIES SCIENCE & ENGINEERING EDUCATION PROJECT June 8, 2004 CURRENCY EQUIVALENTS (Exchange Rate Effective May 26, 2004) Currency Unit = Thai Baht Baht 40.738 = US$ 1.0 US$ 1.0 = Baht 40.738 FISCAL YEAR October 1 September 30 ABBREVIATIONS AND ACRONYMS AusAID - Australian Agency for International Development ICB - International Competitive Bidding ICR - Implementation Completion Report MOE - Ministry of Education MUA - Ministry of University Affairs NBF - Non-Bank Financed NCB - National Competitive Bidding NEC - National Education Commission O&M - Operations and Maintenance PCC - Project Coordinating Committee S&E - Science and Engineering SOE - Statements of Expenditure S&T - Science and Technology TASEAP - Thailand - Australia Science & Engineering Assistance Project USEEP - Universities Science & Enginneering Education Project Vice President: Jemal-ud-din Kassum Country Director Ian C. Porter Sector Manager Christopher J. Thomas Task Team Leader/Task Manager: Omporn Regel THAILAND UNIVERSITIES SCIENCE & ENGINEERING EDUCATION CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 3 5. Major Factors Affecting Implementation and Outcome 6 6. Sustainability 7 7. Bank and Borrower Performance 9 8. Lessons Learned 10 9. Partner Comments 12 10. Additional Information 10 Annex 1. Key Performance Indicators/Log Frame Matrix 20 Annex 2. Project Costs and Financing 22 Annex 3. Economic Costs and Benefits 25 Annex 4. Bank Inputs 26 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 28 Annex 6. Ratings of Bank and Borrower Performance 29 Annex 7. List of Supporting Documents 30 Project ID: P004805 Project Name: TH-UNIVERSITIES SCIENCE & ENG. EDUC Team Leader: Omporn Regel TL Unit: HDNED ICR Type: Core ICR Report Date: June 8, 2004 1. Project Data Name: TH-UNIVERSITIES SCIENCE & ENG. EDUC L/C/TF Number: SCL-41600 Country/Department: THAILAND Region: East Asia and Pacific Region Sector/subsector: Tertiary education (100%) Theme: Education for the knowledge economy (P) KEY DATES Original Revised/Actual PCD: 09/27/1993 Effective: 10/27/1997 10/27/1997 Appraisal: 04/25/1995 MTR: 06/01/2000 03/21/2001 Approval: 05/13/1997 Closing: 12/31/2002 12/31/2003 Borrower/Implementing Agency: GOVT OF THAILAND/MIN. OF UNIVERSITY AFFAIRS Other Partners: Co-financing provided for TA by Government of Australia STAFF Current At Appraisal Vice President: Jemal-ud-din Kassum Javad Khalilzadeh-Shirazi Country Director: Ian C. Porter Javad Khalilzadeh-Shirazi Sector Manager: Christopher J. Thomas Sven Burmester Team Leader at ICR: Omporn Regel William Rees ICR Primary Author: Robert L. Mcgough 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability: L Institutional Development Impact: SU Bank Performance: S Borrower Performance: S QAG (if available) ICR Quality at Entry: Project at Risk at Any Time: No 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The project fully met and, in some cases, exceeded its stated development objectives. The overall objective was to improve the quality of undergraduate science and engineering programs. Specifically, the project sought to: (a) strengthen the teaching capabilities of the faculty; (b) upgrade the content of existing programs in science and engineering and broaden the range of programs relevant to Thailand's technological advancement; (c) modernize laboratories and strengthen their management; and (d) improve the utilization of resources in faculties of engineering and science and establish a system in the large-scale procurement of equipment. The Bank's project objectives were based on a detailed assessment of needs, and the project design was well conceived. The project was prepared with a view toward the reduction of its complexity and risk. A unique aspect of the project was its close articulation with a technical assistance/staff training project that was parallel co-financed by the Government of Australia. This project was financed through the Australian Agency for International Development (AusAID) and was called the Thailand-Australia Science and Engineering Assistance project (TASEAP). TASEAP's objectives complemented those of Universities Science and Engineering Education Project (USEEP) and it effectively supported the agreed upon technical assistance and fellowship/ training programs that supported the objectives of the USEEP. These complementary relationships were originally planned and presented in the USEEP's Staff Appraisal Report (SAR). TASEAP's final expenditures were about US$14.0 million. Additional information regarding TASEAP is provided in later sections as required to clarify the overall impact of the investments made in the USEEP. The final Project Completion Report for TASEAP is listed in Annex 7. 3.2 Revised Objective: The project's development objectives were not revised. This was significant considering the unforeseen Asian Financial Crisis (1997-1998) which resulted in the restructuring of many projects in Thailand. 3.3 Original Components: The project supported quality improvements in 20 public universities through the financing and implementation of the following components. All components were judged to: (a) have strong relevance to the project's objectives; (b) be well within the capacity of the implementing agency; and (c) have been designed to reflect 'lessons learned' from previous Bank projects in Thailand and the Higher Education Sector in general. Component 1. Strengthen the teaching capabilities of faculty - This component was designed to improve the teaching skills and technical knowledge of staff from participating faculties of science and engineering. The project provided international fellowships, in-country training programs and technical assistance by foreign specialists. Small-scale research projects were also financed to extend the knowledge of academic staff relevant to improved teaching. The component also provided English language training for academic and technical staff selected for overseas training and to enable a broadening of their access to academic resources. - 2 - Component 2. Upgrade the content of existing programs in science and engineering and broaden the range of programs relevant to Thailand's technological advancement - This component supported faculties to evaluate existing programs and identify new ones that support the national development priorities. It also supported the improvement of program resource management in the participating institutions. The component provided technical assistance and overseas fellowships and coordinated the development of linkages with international universities and with industry. Component 3. Modernize laboratories and strengthen their management - The project financed equipment procurement to improve the quality of undergraduate science and engineering programs. The component also sought to enhance the capacity of existing institutional staff responsible for management, operations and maintenance. To support this second component initiative, overseas fellowships, training and technical assistance were provided. Included in this component were investments to support the development of the Global Development Learning Network Center at Chulalongkorn University. Component 4. Improve the utilization of resources in the faculties of Engineering & Science and establish a system for the large scale procurement of equipment. In Component 4, there were two elements. The first element provided training and technical assistance to ensure that adequate provision of consumable materials, spare parts, supplies, maintenance, support services and systems for monitoring and analyzing the utilization, operation and maintenance of equipment. The second element provided technical assistance and training to enhance the capacity of the Ministry of University Affairs (MUA) to manage all aspects of large scale procurement under World Bank ICB guidelines. 3.4 Revised Components: The USEEP components were not formally revised although there were some modifications made to their associated Key Performance Indicators (KPIs). The KPIs were revised through a series of meetings between MUA, World Bank and the participating institutions with the intent to more clearly define the deliverables to be achieved under the project. 3.5 Quality at Entry: This project was not subjected to a QAG review and rating at entry. The ICR rating for quality of entry is as satisfactory. This rating is based on a number of findings: (a) from the beginning, the project was strongly focused on its agreed upon developmental objectives. This focus did not waiver or change through the duration of the project; (b) the focus of the project was closely aligned with Royal Thai Government (RTG) higher education development priorities as defined in the 7th and 8th five-year plans. These attuned relationships remained consistent throughout the project's implementation period; (c) the project design was clearly consistent with the Bank's Country Assistance Strategy (CAS); (d) the RTG considers the project to have been timely, in that it occurred in accord with RTG higher education development initiatives and was in place to support the higher education sector during the time of economic crisis in Thailand. As a result the project made a more significant contribution than originally envisaged; (e) the project was clearly designed to leverage the potential of the Bank's investments by closing linking with the technical assistance project funded by AusAid, and (f) the project design was formulated to respond to the a well researched and documented set of "lessons learned" from previous projects in the sector. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: - 3 - The project has met its overall development objectives and its outcomes are seen as substantial. The project supported the Government's efforts to build a comprehensive science and engineering model for achieving better learning at the higher education level. These outcomes were produced by a well-conceived and strongly integrated approach that featured: (a) the training and upgrading of teaching and administrative staff; (b) the upgrading of existing, and the development of new curricula; (c) the establishment of long lasting links with foreign universities, to strengthen research and to facilitate staff exchanges; (d) the development and implementation of a system-wide quality assurance system; and (e) the development and institutionalization of a comprehensive asset management system. The project also demonstrated that a large university system can be changed by targeted investments and well-conceived interventions. The participating institutions are better able to meet the changing demands of the scientific and engineering labor market. All of the participating institutions now report that they have substantially enhanced capacity to produce more science and engineering graduates. Studies have also shown that graduates are now better trained to handle the rapidly changing technologies in the productive sectors. Furthermore, the job placement rates from these institutions continue to be satisfactory. These graduates are recognized as an important contribution to the strengthening of the science and technology infrastructure which is vital to sustaining Thailand's position in technology advancement and global competitiveness. In addition to the above, it should be noted that the each of the project's development objectives was reached within the budget allocated, and within the time periods agreed upon between the Bank and the Government. There was also a sustained Government commitment throughout the project implementation period. Furthermore, the project received strong and committed institutional participation. There was also strong indications of client satisfaction with the implementation and outcomes of the project. Based on these outcomes, the overall outcome of the project is judged to be satisfactory. 4.2 Outputs by components: Component 1: Strengthen the teaching capabilities of faculty - The implementation of Component 1 was satisfactory. Selected teaching staff were provided with short term specialized training to upgrade their teaching skills and technical knowledge - To improve the skills of the teachers in all 36 recipient faculties, the project implemented 139 staff development programs (47 in Engineering and 92 in Science). In each program, technical training, fellowships, research and in some cases the establishment of institutional linkages between Thai and Australian Universities were provided. The project also initiated generic staff development activities in areas that were common within science and engineering. These included: (a) curriculum development; (b) teaching methodologies; (c) research methodologies; (d) library science; and (e) technical support. With the associated support of TASEAP, 2,836 teaching staff of the science faculty, and 805 staff of the engineering faculty were trained in-country. 96 teaching staff of the science faculty and 84 staff of the engineering faculty were trained in Australia. 2,295 participants received training in pedagogy and curriculum development in-country, and an additional 11 persons received such training in Australia. 9,268 participants were also provided with English Language Training (ELT) to enhance the overall ELT capacity within each participating institution. The RTG also provided training for 798 participants to strengthen their capacity to operate and maintain specific items of equipment purchased through the project. Through both TASEAP and the RTG, 368 short-term, specialized overseas fellowships (250 and 118 respectively) were also awarded. Ongoing monitoring and evaluation within the institutions found that the above staff interventions resulted in: (a) improved teaching methodologies such as a reorientation to student-based learning models and a - 4 - greater use of computer-aided instruction including web-based learning; (b) improved capacity to formulate and develop curriculum; (c) greater knowledge of the use and maintenance of equipment; (d) increased research and improved research methodologies; (e) improved access to English language resources; and, (f) improved capacity to develop appropriate resources. One of the most significant indirect benefits of the project was its influence on bringing academics, with like interests, together to share their experiences, knowledge and, in some cases, resources. This interaction also facilitated the development of academic networks in the areas of (a) computing, (b) analytical chemistry, and (c) English Language Training. Component 2: Upgrade the content of existing programs in science and engineering and broaden the range of programs relevant to Thailand's technological advancement - The implementation of Component 2 was satisfactory (for additional data regarding program enhancement/development, see Appendix 1). It is estimated that more than 3,000 engineering and science courses/modules were enhanced or developed. This represents an average increase from base year (1997) of 13.3% in undergraduate and 82.7% in graduate engineering programs, and 24.6% in undergraduate and 49.07% in graduate science programs. TASEAP also provided about 900 person months (or 75 person years) of technical assistance across a range of subjects. This included both long-term (665 person months) and short-term (235 person months) advisors. Component 3: Modernize laboratories and strengthen their management - The implementation of Component 3 was satisfactory. The enormity and complexity of the task is demonstrated by the following statistics: (a) number of equipment specifications written was about 12,217; (b) number of sites to which equipment was delivered was about 295; (c) number of items ordered was about 38,671; and (d) number of tenders was 14. Initially, participating institutions had some concern about the procurement process. Although the procured equipment generally met functional specifications, many universities were not satisfied with the overall quality of the received items. In response to this concern, the Loan Project Office (LPO) established a specifications review committee with representative membership from participating universities. As a result, there was a minimum of further discontent regarding the procurement process. The LPO managed the procurement process effectively and generally, through the duration of the project, employed sufficient dedicated staff to ensure quality operations. There was some new construction and facilities modification as required to house the equipment provided under USEEP. This construction was funded outside of the project by RTG. In particular, newly established universities were allocated RTG budgets to support the construction of new buildings for the faculties of science and engineering. Both the RTG and TASEAP provided substantial training support as mentioned above. TASEAP also provided significant support in the area of laboratory management. A team of TASEAP short-term advisors (STAs) delivered in-country workshops to about 328 academic and laboratory staff in the areas of planning, staffing and operational aspects of laboratory management. The team also provided targeted fellowships for technical support staff to visit Australia and observe laboratory management techniques used in Australian universities. The STAs further assisted in the production of a PC based asset-management system and a multimedia package containing information on relevant standards, calibrations and references. All of these materials were made available to each participant, with a copy sent to the appropriate Dean of Faculty. All of the participating universities now have an asset management program in place. This program is functioning well and it is expected that these universities will fully - 5 - institutionalize the use of the asset management program over the next few years. The RTG sponsored a total of 584 personnel from science and engineering to participate in in-country training courses on operation and maintenance of advanced equipment, while 70 individuals were sponsored for advance training on equipment utilization aboard. Those trainees from both in country and abroad came back and organized training within their faculties. The LPO also confirmed that about 493 additional staff from science and engineering were trained, during the project period, within their own faculties. Component 4: Improve the utilization of resources in the faculties of Engineering & Science and establish a system for the large scale procurement of equipment. The implementation of Component 4 was satisfactory. As mentioned above all procurement has been completed in a satisfactory manner. Furthermore, all of the public universities are now undergoing a new system of budgeting which will incorporate depreciated costs as part of their budgets. This will help each faculty to identify and plan for equipment replacement and future need for both teaching and research functions. An independent review of universities commissioned by the LPO and, undertaken by Kasetsart University confirmed that all of the participating universities now have an established quality assurance system. This system includes the monitoring and evaluation of procedures, equipment usage, and the appropriate use of supplies and instructional materials. The Kasetsart University team also confirmed that each faculty has appointed one or more coordinators for asset management. This indicates that a commitment has been made to ensure that there are no shortages in supplies and instructional materials required for the teaching of science and engineering. Additional information in this regard will soon be available from the ongoing Impact Evaluation Study (funded outside of the project by the RTG). Preliminary findings are expected to be available in September 2004. The project has also initiated a considerable amount of activities to support facility management, and has introduced asset management and maintenance systems. Faculties are continuing the process of developing systems to inventory equipment, supplies and materials. All participating institutions have implemented asset management systems. Faculties have also demonstrated a commitment to budget for necessary instructional supplies and materials to support the undergraduate teaching of science and engineering. Over the last three years (2001 to 2003) science faculties committed a total of Baht 91.427 million to this effort, while engineering faculties have committed a total of Baht 80.504 million. Furthermore, in all cases, the budget significantly increased from 2002 to 2003. 4.3 Net Present Value/Economic rate of return: Not Applicable 4.4 Financial rate of return: Not Applicable 4.5 Institutional development impact: A review of the KPI reveals that significant institutional development impact has already taken place. Within the 20 participating institutions, there has been a substantial increase in the capacity (in most cases exceeding initial project estimates) to: (a) design, plan and implement new courses in science and engineering; (b) upgrade existing courses; (c) plan and implement large scale scientific - 6 - and technical procurement; (d) manage large scale development projects in higher education; (e) operate and maintain sophisticated, technical teaching equipment; (f) establish and maintain institutional linkages (both national and international) to support intellectual exchanges and collaborative research. Additional data and analysis is expected from the RTG's ongoing Impact Evaluation Study (completion is expected in September 2004). 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: The Asian Financial Crisis was a significant event during the implementation of the project. It is interesting to note that the Crisis was judged to have had mostly positive effects on the outcomes of the project. The onset of the Crisis pushed the project to higher levels of visibility and importance within the Government. It became evident that the RTG could not support the continued development of the universities at this scale without USEEP. As a result, the project had significantly more impact on higher education policy reform than originally envisioned. In the initial stages there was some delay in the implementation of the procurement activities. The Government proposed to use an electronic bidding approach to support its procurement requirements. At that time, the Bank had no policy on electronic bidding, so it initially refused to support the proposal. This resulted in some confusion and borrower discontent. Later this decision was reversed. The use of the electronic bidding procedure has proven to have a number of advantages to the borrower, among which the primary advantage was cost effectiveness, thus, it should be considered a 'lesson learned'. In a small number of instances, suppliers did not maintained their contractual commitments relating to ongoing service and the availability of spare parts. Because of this situation, in a few instances, some universities have a few pieces of equipment that are less than fully functional. To address these issues, the LPO initiated meetings between suppliers and the participating universities to discuss their mutual concerns. It was found that there were specific concerns that were common to more than one institution. This recognition and a subsequent dialogue with the vendors led to a better understanding of needs and a stronger adherence to agreements regarding service and spare parts. 5.2 Factors generally subject to government control: There were some government delays in the approval of procurement recommendations. This was partially due to the fact that the project was sizable with a complex procurement plan. There was also an understandable lack of experience with many Bank procurement procedures. Despite the project being one of the first implemented after Thailand's long absence from Bank lending, the counterpart contribution has been satisfactory, and the RTG has demonstrated a strong commitment to the full implementation of a quality project. 5.3 Factors generally subject to implementing agency control: There were some procurement processing delays at the implementation agency level. The following are considered the primary factors that contributed to these delays: (a) there were some delays in the acceptance of goods by respective university appointed acceptance committees; (b) a significant number of LPO staff were under project duration contracts, so as they found more permanent work many resigned leaving a void which took time to fill; and, (c) MUA staff were assigned to the project but often felt that as a result, they lost their opportunity for advancement within the MUA. The MUA became conscious of this issue towards the latter stages of implementation, and moved to establish promotion procedures that effectively addressed this issue. - 7 - 5.4 Costs and financing: Although the project substantially met all its objectives and in some areas exceeded initial expectations, there were significant cost savings. The Government canceled US$11.0 million from the loan on December 6, 2001 and US$3.0 million on October 31, 2001, for a total of US$14.0 million. The project has disbursed about US$127.4 million (about 98% when the loan amount is adjusted to reflect loan cancellations). The cost savings were generally due to a combination of the following factors: (a) lower than expected costs for purchased equipment; (b) the state of the economy during the Asian Financial Crisis; (c) rigorous, centralized specification procedures and associated training that were initially used by the project team; and (d) higher than necessary estimates of equipment costs made during project preparation. Significant variances were found between the estimates at appraisal and actual costs for several categories of N.B.F expenditure. The following briefly describes the variances, and the determined reasons for each: l In the SAR, it was estimated that the RTG would spend about US$24.2 million for equipment procured through NCB and NBF. The data indicates that about US$1.7 million (NBF) was actually spent. This amounts to about 7% of the original estimate for the two categories combined. It was determined that: (a) NCB and NBF procurement estimates at appraisal were too high. Most of the project's equipment requirements consisted of technical equipment that required procurement through ICB, thus, under Bank/RTG agreement, counterpart funding was not required; and (b) the national financial crisis provided for substantial cost savings for NCB and NBF purchased equipment. It should be noted that all planned equipment was satisfactorily procured during the project. l In the SAR, it was estimated that the RTG would spend about US$17.0 million for equipment installation and procurement support costs. The data indicates that about US$1.2 million was actually spent. This amounts to about 7% of the original estimate. It was determined that: (a) due, in part, to the country's financial crisis, the project was able to contract for equipment installation, calibration and even training on equipment usage as a part of the purchase price, thus eliminating the need for additional expenditure in this category. It should be noted that all planned equipment installation was satisfactorily completed during the project. l In the SAR, it was estimated that the RTG would spend about US$57.3 million for equipment O&M costs. The data indicates that about US$5.4 million was actually spent. This amounts to about 9% of the original estimate. It was found that significant efficiencies were gained during the project which reduced the funding requirement for this category of expenditure. Some of the efficiencies were gained from the procurement and implementation of ICT technology and the introduction of strong linkages between the 20 participating institutions. Instead of individual development of O&M programs as planned, the early implementation of ICT made it possible to develop O&M models that could be standardized and replicated in all of the participating institutions, thus providing for significant cost savings. Furthermore, the introduction of extensive technical assistance in the area of management and O&M (provided by the co-financier), provided for efficiency improvements that were translated into further cost savings. It should be noted that all of the planned O&M activities were satisfactorily completed during the project. These activities included: (a) computerized data collection and inventory systems; (b) establishment of institutional O&M committees; (c) establishment of depreciation and replacement standards and procedures; (d) establishment of equipment usage and maintenance recording systems; and (e) establishment of training programs to improve usage, maintenance and collaborative use. - 8 - l In the SAR, in the institutional development category, it was estimated that the RTG would spend about US$0.5 million for consultants, and US$0.1 million for training. It was found that the data indicates that about US$0.1 million and US$0.1 million was spent, respectively. It was found that: (a) the co-financed TA component was more robust than originally planned and, for this reason, there was less need for RTG financed consultants and training; (b) the financial crisis provided for significant cost savings in the costs for both consultants and in-country training; and (c) some of the planned consultant services and training were provided by the vendors as a part of their contractual agreements for equipment purchase and installation. It should be noted that all planned consulting and technical assistance was satisfactorily completed during the implementation of the project. 6. Sustainability 6.1 Rationale for sustainability rating: Staff Development - Quality staff are the primary resource of the participating universities as they move toward institutional autonomy. The participating institutions have worked with USEEP and TASEAP in developing human resources development plans and assessment strategies for their academic and support staff. Each has recognized that there must be continued planning and staff development to ensure that university resources are used efficiently and effectively. This recognition has resulted in significant policy reforms in human resources development at the institutional level. Curriculum Development - As mentioned earlier, significant curriculum development has taken place during the project. The objectives and initiatives for this activity were developed by institutional management and staff with some support from MUA. Every effort was made to closely link the objectives to student/employer needs and national development goals. USEEP/TASEAP assisted in facilitating these developments through technical assistance and some training. As a result, the project has witnessed a strong growth in each institution's capacity to implement curriculum development. With its strong philosophical and technical foundation and the current government policy to place a priority on university curriculum development, there is every expectation that this capacity will continue to develop in the future. Research - USEEP/TASEAP has been successful in raising the awareness of the complementary nature of research and academic development. It is now recognized at both the institutional and MUA levels that quality research is an essential element in all comprehensive university education schemes. Associated with this understanding, MUA has initiated a number of programs to support the development of both undergraduate and postgraduate research at universities with a view toward enhancing the quality of academic programs. Cooperative research programs with international universities have been established. This effort was strengthened directly through the project's and MUA's linkage programs, and many of these have been sustainable to this point. The government's strong commitment to research will likely grow and will almost certainly be sustained for the foreseeable future. Facilities - Financial support for facility management and operation has been flowing as part of the RTG's recurrent funding to universities. It is expected this level of funding will be sustained in the future. In addition to these levels of funding, some faculties are also seeking alternative sources of funding through the provision of services to industry and through short-term commercial training. It should also be noted that through joint research activities developed under the linkages program of TASEAP, Thai universities are now able to access Australian support for operational and maintenance challenges in their laboratories. Australian academics also benefitted from the substantial resources now available at Thai universities. - 9 - University Linkages - It has been the experience of the project that sustainable and effective linkages rely on close relationships between individual academics at each linking university. USEEP and TASEAP have been very successful in supporting such linkages both formally and informally. Although Thai institutions and their international partner institutions have to contend with increasing financial pressures that challenge their efforts to maintain such linkages, this will be countered by both Thai and partner university management's recognition of the need to promote international activities and collaborative arrangements. Procurement Capability - In the initial stages of USEEP, it was expected that the role of the Loan Project Office (LPO) would continue for many years as the Ministry sought additional loan funds for planned sector developments. The RTG has since decided to reduce or eliminate all external borrowing. Therefore, it is expected that the LPO will gradually reduce its capacity following the completion of the USEEP. The LPO will, initially, continue to function as it provides ongoing support to other ongoing Bank project(s). Currently, the LPO staff is providing implementation assistance to the ongoing Secondary Education Quality Improvement Project. Over the long term, it seems reasonable to assume that the LPO's extensive procurement capacity may not be required and that its staff will be eventually returned to their pre-project assignments/positions, most of which are not involved in procurement of goods or services. The project has also provided considerable skills to many young Thai professionals that will enhance their careers in future years. This has already been evidenced by the rate of movement by LPO contract staff into attractive and secure positions outside the MUA. The individual institutions have also enhanced their procurement capacity significantly. Most indicate that they should be able to maintain their current capacity level. 6.2 Transition arrangement to regular operations: Virtually all of the transitions to regular operations have already taken place. 7. Bank and Borrower Performance Bank 7.1 Lending: Although the project's task team initial missions were the first to engage in an education lending activity, in Thailand, in over a decade, it was able to produce a thorough and comprehensive proposal that did not change much during the implementation of the project. A number of working papers were produced that provided significant input to the preparation of the project. The preparation mission paid special attention to the holistic integration of equipment, facilities, pedagogy, curricula and staff development in the project design. Project development was consistent with the Government's long-term plan for university development as well as the Bank's view in this regard. Preparation and appraisal teams were well staffed and managed. Extensive data collection took place and the results were well organized and documented. Lessons learned from other projects were discussed and agreed upon. The appraisal of the project was comprehensive. The performance indicators were well prepared and continuously updated, making the tracking of the project activities straight forward. Delays in project negotiation and effectiveness were due to the Government's decision to slow down the process for internal reasons. The task team's initial project estimates for equipment costs may have been somewhat excessive, however, this is difficult to confirm because the project was implemented during a financial crisis in Thailand and there were other contributing factors as well. Overall, the Bank's performance in this section was satisfactory. 7.2 Supervision: The Banks files contain full documentation of all PSRs and there is evidence of extensive Bank involvement as issues were identified during the project's implementation. Initially, similar to many other projects, the progress of the project was sluggish. Throughout this initial period there was close monitoring and - 10 - supervision by the Bank. As a result, the early issues (primarily associated with procurement management) were quickly identified and addressed. Through the Bank's intensive and pro-active supervision, implementation progress was adequate reported and project performance was realistically rated. All covenants were reviewed during each mission and strictly enforced. Staff gave significant advice and showed considerable flexibility in solving implementation issues. Supervision missions at six month intervals were adequately staffed. Relations with the Borrower were excellent throughout the project period. It is noteworthy to mention that most of the supervision was conducted by senior staff from the Bank's Bangkok Field Office. This arrangement seems to have resulted in a closer communication and oversight than would have been typically found if the project was supervised from headquarters. Overall, the Bank's performance in this section was satisfactory. 7.3 Overall Bank performance: The overall Bank performance was satisfactory. Borrower 7.4 Preparation: The Borrower assigned a counterpart team to actively participate in the identification and preparation of the project. Senior government officials made themselves available for discussion and policy dialogue during all phases of the project. They made many significant contributions to the soundness of the concept and scope of the project. Likewise, the participating institutions showed a strong commitment to excellence, and a full implementation of project. The overall performance of the Borrower during project preparation was highly satisfactory. 7.5 Government implementation performance: The Government was, and remains, strongly committed to the higher education sector. Over the past decade the participating universities have gained increasing prominence and capacity to contribute to the national development agenda. For these reasons, it is expected that the Government will continue to support the funding required for the operations and development of higher education. Throughout the duration of the project counterpart funds were available and adequate. Covenants were in full compliance and there were no overdue audit reports through the project years. The overall performance of the Borrower during implementation was satisfactory. 7.6 Implementing Agency: The implementing agency was strongly committed and involved with all aspects of the project throughout the duration of the project. Initially, the implementing agency attempted to implement the project using a strong central control concept. There was some concern within the participating institutions that this approach was less than satisfactory because, in their opinion, the implementing agency could not provide adequate oversight to the widely disbursed activities of the project. As these issues were brought up, the implementing agency responded well with demonstrated flexibility. Changes were made in the implementation plan to allow for a more decentralized approach wherein, the various institutions became responsible for many aspects of the project's implementation. This approach proved to be highly satisfactory to all parties. The Bank's supervision team was included in the related discussions and were fully consulted as the changes were planned and made. The overall performance of the implementing agency was satisfactory. 7.7 Overall Borrower performance: Overall the Borrower's performance was satisfactory. - 11 - 8. Lessons Learned The use of local processing staff based in the Bank's field office is an advantage for projects which require frequent monitoring and evaluation. In the case of this project, the LPO made frequent use of Bank advice that was locally available on procurement, governance issues, project financing and development strategies. A clear orientation to the project objectives and a statement of project responsibilities should be provided to all participants in a project. Initially the project team did not fully orient institutional staff to the project's objectives and their respective roles in implementation of the project. Subsequently, there was some confusion and misdirection in the early days of the project. Upon realizing the problem, the project implementation unit moved quickly to address the problem. Project designs should include planned activities to quickly collect and organize good baseline data. Some baseline data was initially available for enrollments, academic qualifications and courses, however, other qualitative data on student development was not collected until much later. Furthermore, the important monitoring and evaluation plan for USEEP was not completed until midway through the project. It should be noted that its completion met the conditions of the project, however, in retrospect, it would have more useful if completed earlier. When co-financed projects are associated with a Bank project, it is important to have clear understandings and linkages/relationships well documented for both projects. TASEAP was a linked-project with USEEP. It focused on the first two components of staff and academic program development, and part of component 3, in that it supported the establishment and ongoing operation of a procurement capacity within the implementing agency. Yet, because TASEAP was funded by a partner government (Australia), it was treated as a separate project which was complementary to USEEP, but not included as part of the overall USEEP project implementation plan. Where faculties have similar or related academic interests, careful attention should be given to options for integration and sharing of equipment and other resources. Initially, the project found it difficult to gain faculty recognition of this need. Each faculty had a natural inclination to seek resources to fund their independent teaching and research capacity. Later, a MUA review of this issue revealed that there were a number of overlaps and inconsistencies in processes, documentation and required resources for similar programs in both the Science faculty and the Engineering faculty. From then on, there was an effort to seek significant integration of both Engineering and Science streams, for example in areas such as generic teaching and research, (i.e. common to both streams). This integration resulted in a more efficient use of resources. Future projects should not assume that there is good communication within its stakeholder organizations and should evaluate and work with these organizations to ensure all interested parties can participate in information flows. The project did have a comprehensive communication strategy that was approved and disseminated to all stakeholders. However, the project overestimated the effectiveness of communication within the universities and faculties. Some problems occurred when communication broke down between participating institutions and/or the LPO. Where only one language is predominantly spoken in a project area, that language should be the primary language used in the project. There was an assumption that many, if not most, of the academics participating in the project would have a good understanding of English, a second language. In many instances this was not the case, and there was some miscommunication as a result. The strong management within the respective institutions usually rectified these misunderstandings, but this situation did cause - 12 - some delays and staff confusion. Electronic bidding procedures have been proven in this project to have distinct advantages over traditional procurement/bidding procedures. 9. Partner Comments (a) Borrower/implementing agency: Background and Rationale In late 1993, the Royal Thai Government requested The World Bank to examine the possibility of project assistance to education. In conjunction with the Government, the Bank identified a project which would assist in strengthening the quality of science and engineering education in public universities. Most of the preparation work was completed by the Government according to guidelines formulated by the Bank. Project preparation was completed in November 1994 and the project was appraised in May 1995. Project processing was delayed following elections in July 1995 and installation of a new Government. Further delay was caused by uncertainties and could be resolved, regarding the provision of technical assistance. Finally, the project was signed on July 29, 1997. The project was consistent with the Government's priority for developing technical and scientific skills and with its plans for expanding and improving the quality of science and engineering education. The project also emphasized on human resource development and the need to develop the capacities of the universities to provide quality science and engineering education. The overall objective of the project was to improve the quality of undergraduate science and engineering programs. The specific aims were to; strengthen the teaching capacities of faculty, upgrade the content of existing programs in science and engineering and broaden the range of programs relevant to Thailand's technological advancement, modernize laboratories and strengthen their management, improve the utilization of resources in faculties of engineering and science, and establish a system for the large-scale procurement of equipment. The project supported quality improvement in 20 public universities through financing an institutional program including; short-term overseas training for selected academic and technical support staff in the education use and maintenance of the project-finance equipment and technical assistance and academic support services which was financed by the Australian Agency for International Development through Thailand- Australia Science and Engineering Assistance Project (TASEAP). The TASEAP also provided consultant services to evaluate existing programs and reconfirmed new ones, improved the management of program resources, developed effective procedures for the large-scale procurement of equipment and strengthen project management. The component included the financing of small-scale research projects which extended the knowledge of academic staff relevant to improve teaching and English language training for academic and technical staff selected for overseas training. Project Implementation The Loan Project Office, Ministry of University Affairs was established to be responsible for centralized equipment procurement, planning and management of overseas training programs and in-country training, coordination of consultant services, monitoring and evaluation, financial management and information dissemination. Co-ordination of the project with co-financier was made through the Department of Technical assistance Cooperation. Coordination, monitoring and evaluation of Australian contribution to the project was undertaken by the Australia for International Development through its office in Canberra and Bangkok. The Australian Managing Contractor responsible for - 13 - project implementation was SAGRIC International Pty. Ltd. in association with Monash University, the University of New South Wales and the University of South Australia. The project key performance indicators were developed and modified continuously during the project implementation period by every part of the project concerned. After the fifth year of project implementation, the government requested the Bank for a one-year extension since there were many uncertainties occurred causing the project timely completion as identified in the contract. At the same time, Kasetsart University research team was contracted by the Loan Project Office to conduct further monitoring and evaluation of the project. Evaluation of Performance against Objectives Objective 1 Strengthening teaching capability Academic development activities in science were targeted at nine undergraduate priority areas. These included analytical chemistry, biotechnology, computing, ecology/biology, environmental science, material science, organic chemistry, physics and statistics. There were over 140 in-country training workshops were carried out in these priority areas. These were attended by over 2400 Thai academics from across all the participating universities. In addition, the project provided 96 fellowships to Australia which enabled Thai academics to further study specialist academic and research areas with their Australian colleagues. The RTG had supported specialist in country training for about 543 Thai academics and technical support personnel, on specific items of science equipment purchased by faculties. It had also sponsored international training to a further 47 science academics, covering specific operations and maintenance on some of the more complex equipment purchased. As in science, engineering activities were targeted at priority areas for undergraduate academic development. These consisted of construction engineering, energy engineering, environmental engineering, machine design, material processing, electronics, telecommunications and transport engineering. Over 40 in-country workshops were carried out in support of these priority areas, which were attended by over 800 academic staff. In addition, the project provided 84 fellowships to Australia for Thai engineering academics. The project counterpart fund of RTG also supported 319 participants for in-country training courses of Thai academic and technical staff on the operation and maintenance of specific items of engineering equipment. It also sponsored international training to a further 23 engineering academics on specific interests in operation and maintenance of some of the more complex engineering equipment purchased. Objective 2 Upgrade existing undergraduate programs The provision of new equipment through the project enabled faculties to both establish new courses, in cases where existing facilities were inadequate, and improve existing curriculum, particularly in terms of associated laboratory work. The universities have identified a total of 2,005 courses in science and 991 coursess in engineering have been adjusted as a direct impact of the project. There were 1,084 courses in science and 244 courses in engineering developed. Three hundred and five programs in science and three hundred and sixteen programs in engineering were developed in both undergraduate and graduate levels. The project has provided faculties with a range of new equipment to enhance the way subjects are taught. The greater use of technology now being applied to the teaching of science and engineering - 14 - programs. The project provided direct technical assistance to faculties in regard to developing improved teaching methodologies. This was provided broadly through a series of in-country workshops, attended by a total of 947 academics. The project also provided specific support direct to individual faculties in the application of newly acquired equipment in their teaching approaches. Although the project was targeted at the development of undergraduate programs, a secondary benefit of the provision of equipment has been to enable the further development of research within participating faculties. Faculty staff and postgraduate students have now greater access to modern scientific and engineering equipment which they can apply to their research programs. The participating universities have identified a total of 1,991 new research projects which have resulted from this new access. In addition, the project initiated a program, which was targeted at raising the awareness of Thai academics of the complementary nature of research and academic development. In addition to the provision of technical assistance direct to faculties, the project provided 5 workshops on the improved research methodologies. The report of the contracted team noted that the project also provided opportunity for faculties to strengthening their relationships with other international universities. It sponsored, both directly and indirectly, the development of 68 such linkages. These were in the areas of joint Doctoral, Master of Bachelor programs, research, teaching and student exchange programs. Objective 3 Modernize laboratories and strengthen their management In addition to the training of academic and technical staff in the specific use and maintenance of equipment afforded through the project, training was also provided in the area of laboratory management. The project offered six fellowships for technical support staff to visit Australia and observe laboratory management techniques used in Australian universities. These staff were then involved in a series of in-country workshops in which they relayed their findings to the colleagues from other universities. The project promoted and supported the development of asset management and asset utilization systems within faculties to manage and monitor the existing and new equipment. This included the provision of asset management software and database. Every faculty has made further advances to the development of such a system which has enhanced the sustainability of equipment provided through the project. At the same time, 79 standard laboratories in science and 68 standard laboratories in engineering were developed for teaching, research and services. The English language program was made successfully to enhance existing English language activities at universities in the areas of science and engineering. The project provided six fellowships in Australia for English language academics who produced a computer assisted language learning software package and support materials specifically targeted at science and engineering. These fellows then returned to Thailand and participated in project sponsored workshops where this new software package and associated support materials were presented to English language staff from all participating universities. Objective 4 Improve utilization of resources in the faculties of science and engineering The procurement of equipment was completed with 38,671 pieces of fourteen tenders as identified - 15 - in the contract and has been accepted and installed to the sites. The study of equipment utilization conducted by the contracted team revealed that 80.85 percent of science equipment and 70.76 percent of engineering equipment was used for teaching per week. In addition, 12.89 percent of science equipment and 15.29 of engineering equipment was used for research and 6.26 and 13.95 percent was used for services per week. During the implementation period (2001-2004) 253 millions Baht of counterpart fund was allocated directly to the universities for operation and maintenance cost of the equipment provided. Chulalongkorn University ­ Global Development Learning Network : CU-GDLN As part of the Universities Science and Engineering Education Project : USEEP, CU-GDLN aimed to promote student and faculty development in higher education institutions throughout Thailand and serve the government and private sectors by sharing and exchanging knowledge and resources on the issues related to national interest. CU-GDLN offers different levels of videoconferencing service for distance education, training, seminar and tele-working to serve both low-end and high-end markets. Linking to the World Bank's GDLN through the Network Operation Center at the Bank headquarter in Washington, D.C., the CU-GDLN uses three channels for the linkups; satellites, ISDN telephone line, and the Internet. The network provides videoconferencing at transmission speeds of 128-384 kilobit per second (kbps) and network speeds of 256 kbps so conferences can be broadcast to all Distance Learning Centers live audio-visually worldwide. More than six hundred thousand USD has already been disbursed since the year 2000 for the equipment procured including operation and maintenance costs. The World Bank's representatives and the Loan Project Office`s staff visited CU-GDLN periodically for monitoring and evaluation of the USEEP during the project implementation. The results of continuous visits were as follows; 1.Management CU-GDLN has been managed as a part of Chulalongkorn University Center of Academic Resources : CAR. Seven staff of the CAR were assigned to work on temporary basis on general administration, training coordination and technical works under the supervision of a senior advisor, (former director of to Thailand Document Center, Chulalongkorn University). With their expertise on their own jobs at CAR, all staff are able to work effectively and efficiently. 2.Performance Started in October 2002, CU-GDLN organized 24 events with 82 sessions for 871 participants in 2002 of Thai fiscal year and 26 events with 59 sessions for 839 participants in 2003. The participants were university students, lecturers, government and non-government officials. The diversity and applicability of contents offered with powerful technology used for real time training were highly satisfied by the participants. 3. Sustainability CU-GDLN was developed on academic and technological potential of Chulalongkorn University with two projects; WB-CU Documentation Center and the WB-CU Digital and Online Information System. The above mentioned projects are connected to University Networks: UniNet and served as a super hi-way to higher education institutions throughout Thailand. Knowledge exchanging and sharing can be done both within country and globally by the network provided. - 16 - In the near future, CU-GDLN decided to develop courses and contents for worldwide participants in order to act as an information and knowledge provider in the information-based society. 4. Strenthening the Sustainability Development Although CU-GDLN has been managing smoothly with concrete support from Chulalongkorn University and the World Bank in term of facilities and expertise, the Center still needs continuous supports on long-term finance and full time qualified personnel for sustainable development. In addition the Center itself should strengthen its Strategic plan and Financial plan which has to be incorporated in the university plan and follow higher education development plan to be fully supported from the Budget Bureau. Furthermore internal management should be focus more on the following activities 4.1 Course Development and Delivering should be more closely link to global and local needs and has to be fully supported by the government as an alternative ways of HRD. 4.2 Increase greater public awareness of the role and activities of the Center 4.3 Moving toward the adoption of Business oriented management in order to increase efficiency in term of financial management. Major Factors Contributing to the Success of the Project The project was one of the key supporting mechanisms that helped sustain the development in science and engineering education in Thailand and was well accepted by most universities especially those newly established universities. The project provided faculties of science and engineering a considerable kick-start with respect to laboratories management. In particular, Asset Management System initiated by the Project and was established in the faculties of Science and Engineering will be a stepping stone for the new budgeting system of the university. In addition, appropriate training courses, were offered by both TASEAP and USEEP in laboratory management, maintenance and operation of highly sophisticated equipment. As the courses aimed at daily operation and maintenance management of the laboratories and equipment usage, the technical staff brought back their knowledge from the courses provided for the improvement of their own laboratories at the faculties. The project also supported the successful implementation of Higher Education Development Plan in various aspects such as strengthening linkages between universities and industry, cooperated research between public and private sectors, upgraded research capacity of universities, helped develop postgraduate education and supported quality assurance of higher education. It could not be deniable that the success of the project was due to full support from the Government of Australia through his generosity in providing assistance for the academic development for the faculties of Science and Engineering and for procurement management. MUA has also recorded the full support from the World Bank officers, both at Washington D.C. and in Bangkok Office, and recognized its assistance as a major contribution toward the success of the project. Advises from the World Bank always came in time and help alleviate many difficulties we encountered. With effective advice the unused loan was returned immediately instead of delay until the final stage of the contract. In addition our sincere appreciation was going to Department of Technical and Economic Cooperation for her assistance on TASEAP and to the Budget Bureau for a generous allocation of budget for the management of the project and for training and maintenance of the equipment procured. - 17 - The Public Debt Department and the Comptroller office, Ministry of Finance could not be neglected for their kind help and support. Conclusion and Lesson Learned General Conclusion USSEP was the first largest project on development of undergraduate science and engineering programs in public universities. Although the project faced many challenges due to its scope and complexity the targeted outcomes were achieved. The project has also impacted on raising the quality of undergraduate programs in science and engineering and the development of future programs and internationalization of Thai higher education system as it created and initiative for networking among Thai public universities and collaboration with Australian universities. Knowledge gained from the project on procurement system, international bidding, scientific equipment specifications, data based for procurement management system are valuable knowledge for the future implementation of the similar project. Best practices in the forms of web-based asset management, electronic logbook, laboratory maintenance and operation including interactive teaching and learning with problem based learning were brought to the sustainable development successfully since all of the practices mentioned have been implementing in every university concerned. In achieving such complex and challenging project as already mentioned there are many lessons learned from the project such as: Lesson Learned on Management There are 2 aspects regarding the lesson learned on management first the Organization Design and Communication and Involvement Organization Design Even though the Organization of the Implementing Agency was well designed, with help from AusAIDS who provided technical assistance in both academic development and the management of the procurement, it is still found that the full time Thai counterpart with more experience in the area of scientific equipment and Bank financed Procurement system is still very necessary at the beginning of the project. Most of junior staff were temporary employed and hence the project faced frequent staff turnover. In addition the discontinuity of management system at all levels ranging from MUA down to departmental level at the universities. In particular, the change of top management such as the Minister, the Permanent Secretary, the University Presidents, the Deans and the Heads of Departments have effected significantly on the operation of the project. Due to the large size and complexity of the project, Project Implementation Sub Unit (Sub-PIU) should be established at the university level. The Sub-PIU will oversee the project as a whole and will be the suitable point of contact. The benefit of Sub-PIU is that knowledge and experiences gained at LPO could be transferred directly to Sub-PIU. In addition , any problems occurred at a particular university could be solved through the Sub-PIU in connection with LPO which could speed up the whole process. Communication and involvement Due to size and complexity of this Project as mentioned earlier, it was difficult enough to make - 18 - universities fully understand the USEEP and TASEAP projects and the connection of the two projects in early implementation. The difficulty is that the position of the Dean and Department Heads is changed every four years or less in some universities while the Project is over 5 years. Changing of the executive teams lead to changing of coordinators and not all the information had been passed through. The workshop held by LPO each year in order to disseminate information to universities created mutual understanding of the 2 projects. Furthermore LPO published a newsletter sent out to each department heads and coordinators to inform about the updated project events. Regardless of all the effort LPO had done in the communication, the problem was inevitably. There should be a more comprehensive communication strategy to disseminate to all stakeholders with regarding to different perception and process of information flows within the university. With better communication, involvement of stakeholders in all aspects of the procurement cycle should also be taken into consideration. Lesson Learned on Procurement Complexity of the procurement process Part of the problems in equipment acceptance is due to the complexity of the procurement process which was new and take quite a long time. Since the project funding is around US$ 130 million, procurement has to be done in large packages to reduce handling cost. However, such process in turn created additional cost and excess paper work for companies bidding for the contract and created delay in delivery process. Moreover large packages procurement needs to simplify the specification of same type of equipment into one standard, omitting many detail specification which cater to each user's needs and sometimes clarification has to be made at acceptance process . Large Lot At the beginning of the project, large number of equipment to be procured and a lot of equipment varieties in each tender had caused bidders to submit bid outside their scope of expertise and hence they do not have enough technical knowledge of all equipment both in installation and operations. This in turn caused may problems in some of the early tenders such as; delay in equipment acceptance, incomplete installation, ineffective training. Large size tender also has hidden cost in equipment delivery, handling and installation. Vendors Thai Vendors Many Thai vendors did not understand the process of workflow inside LPO and view LPO as part of the government bureaucratic system. Having little experience in international bidding (for many this was their first opportunity for international bidding), many venders had to rely on their sub-contractors for equipment installation and deliver, causing problem and delay in equipment acceptance. Foreign Vendors Some foreign Vendors were not ready and well prepared when awarded multiple contracts or big contracts with many items and hence delayed both delivery and acceptance. In addition, their Thai representatives were not well qualified to perform proper installation or training that created some dissatisfactions to the acceptance committees. - 19 - Mutual Understanding of the Bank Guidelines for Procurement For future project better preparation and better understanding of the Bank Guidelines for procurement of all participating agencies at the early stage of the loan implementation will certainly be one of the key success. (b) Cofinanciers: The Government of Australia, a cofinancier of this project, did not submit comments for inclusion in this ICR. The Government of Australia provided a separate Project Completion Report for its components. That report is listed in Annex 7, and information from the AusAID Project Completion Report was used to support the findings of this ICR. (c) Other partners (NGOs/private sector): 10. Additional Information Not Applicable - 20 - Annex 1. Key Performance Indicators/Log Frame Matrix Output Indicators: 1 Indicator/Matrix Projected in last PSR Actual/Latest Estimate 1.1 Selected teaching staff will be provided 2836 teaching staff of science faculty and In addition to projected in last PSR, 9268 with short-term, specialized training to 805 of engineering were trained; 96 staff of participants were trained in English upgrade their teaching skills and technical science faculty and 84 staff from engineering Language Training (ELT); and 11 knowledge. faculty were trained in Australia; 2,295 participants were trained, in Australia in teaching staff were also trained in-country in pedagogy and curriculum development. pedagogy and curriculum development in 9,268 participants were also provided with Australia; (ELT). The RTG also provided training for 798 participants to strengthen their capacity to operate and maintain equipment. Through both TASEAP and the RTG, 368 short-term, specialized overseas fellowships (250 and 118 respectively) were also awarded. 2 .1 Based on the figures for the base-line 157 undergraduate programs of science were Same as projected in last PSR year, the number of science and engineering upgrades/developed (24.6% increase from courses will be upgraded by an average of at base year (126)); 161 graduate programs of least 20% in the participating institutions. science were enhanced/developed (49.07% increase from base year (108)); and, 1,443 courses of science were upgraded/developed; 179 undergraduate programs of engineering enhanced/developed (13.3% increase from the base year (158)); 137 graduate programs of engineering were enhanced/developed (82.67% increase from base year (75)); and, 1,235 courses of engineering were enhanced/developed. 2.2 Throughout the life of the project, 81 Australian short-term consultants visited Same as projected in last PSR AusAID will provide US$7.7 million to employ and organized 193 training workshops for consultants to assist participating universities academic staff of science and engineering in their development of new, and faculties; 21 Australian short-term improvement of existing, undergraduate consultants visited and organized 131 programs in science and engineering. training workshops on teaching methods and curriculum development; 23 Australian short-term consultants visited and conducted 26 workshops on linkages for joint program collaboration, research, student exchanges and networks. 3.1 Equipment is provided according to an Revised procurement plan is 97.83% Based upon a revised and agreed upon agreed upon schedule of implementation. complete. 37,860 items of equipment and schedule of implementation , 100 percent of supplies were accepted and integrated into equipment and supplies were accepted (The the participating institutional inventories. final count of accepted items was 38,671) and integrated into the participating institutional inventories 3.2 Construction and facilities modification 100% complete is completed according to an agreed upon schedule of implementation (already under construction and financed outside of the project) 3.3 Specialized training will be provided to 835 academic staff and technicians of TASEAP supported in-country training for selected laboratory managers and technical science and engineering faculties were 328 staff in areas of planning staffing and staff. trained in-country and 70 academic staff operational management of laboratories, and were sent for advanced training courses in about 70 staff for overseas training; RTG equipment utilization in Australia. provided in-country training for about 584 staff in operation and maintenance of advanced equipment, and about 80 staff for overseas training in similar areas. An additional 493 staff were provided with in-service training in equipment operations and maintenance 4.1 By the end of the project, all equipment 100% complete - 21 - and associated supplies and materials required by participating universities in undergraduate science and engineering programs will be identified, specified and procured, 4.2 Within 12 months of completion of the Although it is not yet 12 months past project project, the instructional programs will not be completion, instructional programs are not experiencing significant shortages in reporting any significant shortages of equipment, supplies and instructional supplies and equipment. materials procured under USEEP. 4.3 All equipment, supplies and materials 30 faculties modified original asset Same as projected in PSR procured will also be inventoried. management program for local use and applications. 36 faculties recorded equipment utilization in schedules and log books provided for that purpose 4.4 By the end of the project, all relevant and At time of PSR, no response possible All relevant and operative laboratories are operative laboratories will be fully provided fully provided with necessary instructional with the necessary instructional supplies and supplies and materials to support materials to support the undergraduate undergraduate science and engineering science and engineering curriculum being curriculum being taught. taught. - 22 - Annex 2. Project Costs and Financing Project Cost by Component (in US$ million equivalent) Appraisal Actual/ Latest Percentage of Estimate Estimate Appraisal Project Cost by Component US$ million US$ million Strengthening the teaching capability of 4.1 2.6 63.4 faculty Upgrading the content of existing 13.2 12.8 97.0 programs in science and engineering and broadening the range of programs relevant to Thailand's technological advancement Modernizing laboratories and 184.6 130.3 70.6 strengthening their management Improving the utilization of resources in 59.6 7.3 12.2 faculties of engineering and science and establishing a system for the large-scale procurement of equipment Total Project Cost 261.5 1/ 153.0 Total Financing Required 261.5 153.0 1/ Although the project substantially met all its objectives and in some areas exceeded initial expectations, there was significant cost savings. The Government canceled US$14.0 million in 2001. The cost savings were due to a combination of the following factors: a) lower than expected costs for purchased equipment; b) the state of the economy during the Asian Financial Crisis; c) rigorous, centralized specification procedures and associated training that were initially used by the project team; and d) higher than necessary estimates of equipment costs made during project preparation. - 23 - Project Cost by Procurement Arrangements (Actual/ Latest Estimate) (US$ million equivalent) Expenditure Category Procurement Method ICB NCB Other N.B.F. Total Cost 1. Equipment 142.50 16.80 0.00 8.70 167.60 (127.4) (0.0) (0.00) (1.7) (129.1) 2. Equipment O & M 0.00 0.00 0.00 57.30 57.30 (0.00) (0.00) (0.00) (5.40) (5.40) 3. Equipment Installation and 0.00 0.00 0.00 17.00 17.00 Procurement Supporting Cost (0.00) (0.00) (0.00) (1.20) (1.20) 4. Consultants 0.00 0.00 0.00 9.50 9.50 (0.00) (0.00) (0.00) (9.10) (9.10) 5. Training 0.00 0.00 0.00 4.10 4.10 (0.00) (0.00) (0.00) (2.60) (2.60) 6. Universities Linkages 0.00 0.00 0.00 3.70 3.70 (Research Program) (0.00) (0.00) (0.00) (3.70) (3.70) 7. Project Mangement 0.00 0.00 0.00 2.30 2.30 (0.00) (0.00) (0.00) (1.90) (1.90) Total 142.50 16.80 0.00 102.20 261.50 (127.40) (0.00) (0.00) (25.60) (153.0) Remark: Figures in parenthesis are the respective amounts financed by the Bank Loan. All cost include contingencies. N.B.F. : Not Bank - financed - 24 - Project Financing by Category of Expenditure (In US$ million equivalent) Appraisal Estimate Actual/Latest Estimate Percentage of Appraisal Category of Expenditure Bank Govt. CoF. Bank Govt. CoF. Bank . Govt. CoF. 1. Equipment 143.40 24.20 - 127.40 1.70 - 88.84 7.02 - 2. Equipment Installation and - 17.00 - - 1.20 - - 7.06 - Procurement Supporting Cost 3. Equipment O & M - 57.30 - - 5.40 - - 9.42 - 4. Institutional Development 1) Consultants - 0.50 9.00 - 0.10 9.00 - 20.00 100.00 2) Training - 2.60 1.50 - 1.10 1.50 - 42.31 100.00 3) Universities Linkages - 0.20 3.50 - 0.20 3.50 - 100.00 100.00 5. Project Management - 2.30 - - 1.90 - - 82.61 - Total 143.40 104.10 14.00 127.40 11.80 14.00 88.84 11.14 100.00 Remark * USEEP has cancelled loan amount USD 14.00 million,then the existing loan is USD 129.40 million - 25 - Annex 3. Economic Costs and Benefits No economic costs and benefit analysis was carried out at the time of project preparation, nor was it done at the time of the ICR. - 26 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, 1 FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation 11/01/1994 5 Principal Economist (1); S S Engineering Educator (1); Science Educator (1); Technical Educator (1); Operations Officer (1) Appraisal/Negotiation 05/01/1995 5 Principal Economist (1); S S Engineering Educator (1); Science Educator (1); Technical Educator (1); Operations Officer (1) Supervision 08/28/1997 4 MISSION LEADER (1); S S OPERATIONS OFFICER (2); PROCUREMENT SPECIALIST (1) 12/05/1997 4 ECONOMIST (1); S S OPERATIONS OFFICER (2); PROCUREMENT SPECIALIST (1) 06/19/1998 3 OPERATIONS OFFICER (1); S S PROCUREMENT SPECIALIST (1); TECHNOLOGY SPECIALIST (1) 12/10/1998 2 OPERATIONS OFFICER (1); S S PROCUREMENT OFFICER (1) 06/09/1999 7 OPERATIONS OFFICER (1); S S PROCUREMENT OFFICER (1); FINANCIAL SPECIALIST (1); EDUCATION ANALYST (1); DEPUTY PS OF MUA (1); PROGRAM MANAGER AUSAID (1); EDUCATION MANAGER (1) 06/12/2000 3 OPERATIONS OFFICER (1); S S PROCUREMENT OFFICER (1); RESEARCH ASSISTANT (1) 3/01/2001 5 Senior Operations Officer (1); S S Procurement Specialist (1); Financial Specialist (1); Science Education Specialist (1); - 27 - Research Assistant (1) 02/01/2002 5 Senior Operations Officer (1); S S Procurement Specialist (1); Financial Specialist (1); Science Education Specialist (1); Research Assistant 05/01/2003 7 Senior Operations Officer (1); S S Education Specialist (1); Procurement Specialist (1); Financial Specialist (1); Portfolio Coordinator (1); Program Assistant (1); Research Assistant (1) ICR 10/01/2003 5 Senior Operations Officer S S (1); Procurement Specialist (1); Financial Specialist (1); Procurement Analyst (1); Research Analyst (1 (b) Staff: Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation NA 147,906.11 Appraisal/Negotiation NA 73,953.06 Supervision NA 342,782.36 ICR NA 12,124.94 Total NA 576,766.47 - 28 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA - 29 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU - 30 - Annex 7. List of Supporting Documents "USEEP - Project Completion Report for Loan 4160", Loan Project Office, Commission on Higher Education, Ministry of Education, Kingdom of Thailand, January 2004 (Government's Final Report). "USEEP - Progress Report for Loan 4160", Loan Project Office, Commission on Higher Education, Ministry of Education, Kingdom of Thailand, January 2004. "USEEP - Monitoring and Evaluation Report for Loan 4160", Loan Project Office, Commission on Higher Education, Ministry of Education, Kingdom of Thailand, August 2003. Project Communication Files (Aide Memoires and other forms of communication) for Loan 4160, The World Bank; Washington and Bangkok. Project Status Report Files for Loan 4160, The World Bank, Washington and Bangkok. "TASEAP - Project Completion Report", Australian Agency for International Development and Kingdom of Thailand, Ministry of University Affairs, Bangkok, June 2001 (draft). "Loan Agreement Loan Number 4160-TH - Universities Science and Engineering Education Project", between Kingdom of Thailand and The World Bank, July 29, 1997 'Staff Appraisal Report, (Report Number 16151-TH; Loan Number 4160-TH) - Universities Science and Engineering Education Project", The World Bank, Washington, February 7, 1997. - 31 - - 32 -