ICRR 13468 Report Number : ICRR13468 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/11/2011 PROJ ID : P075502 Appraisal Actual Project Name : Ozone Layer US$M ): Project Costs (US$M): 17.00 16.47 Protection Project Country : Venezuela Loan/ US$M ): Loan /Credit (US$M): 0 0 Sector Board : ENV Cofinancing (US$M ): US$M): 0 0 Sector (s): Other industry (50%) Central government administration (50%) Theme (s): Other environment and natural resources management (100% - P) L/C Number : Board Approval Date : 04/30/2005 Partners involved : UNIDO Closing Date : 06/30/2009 12/31/2009 Evaluator : Panel Reviewer : Group Manager : Group : Richard C. Worden Robert Mark Lacey IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: Paragraph 17 of the Memorandum and Recommendation to the Regional VP for LCR ("Director's Memo") states the PDO as: "a) Gradual Phase Out of CFC-11/12 by December 31, 2006, of all production of ozone-depleting substances (ODS), such as CFC-11 and CFC-12, in Venezuela under the Montreal Protocol through a compensation program accompanied by enforcing institutional and legal mechanisms; and b ) a chiller replacement program to set up a national system to replace the country's nearly 300 inefficient CFC-based chillers with energy-saving non-CFC-based units in a cost-effective manner." However, the statements of objectives in the two separate Grant Agreements (GA) signed for this project differ significantly from that in the Director ’s Memo. The GA signed between the Bank and PRODUVEN, the country's only OS-causing CFC production facility (which also produces HCFC-22, a non-ODS CFC alternative), stated the first PDO was “to phase out, by December 31, 2006 , all production of CFC -11 and CFC -12 in Venezuela, � together 31 , 2006, with the “dismantling and retrofitting of all ODS CFC production equipment (reactors, storage, pumps, gauges ) and safe disposal of equipment and stocks � at PRODUVEN (paragraphs 1 and 2 under Schedule 2 of the GA). This was to be accomplished through a compensation scheme to address lost profits and employment at PRODUVEN . The second GA covered the provision of technical assistance (TA) to FONDOIN, empowered by the Government of Venezuela (GOV) to act as its National Ozone Unit. The objective of the TA was to enable FONDOIN "to monitor and enforce the institutional and legal mechanisms to control any renewal of ODS -causing CFC production in Venezuela as well as to support the chiller replacement program � (Part A and B under Schedule 2). The two PDOs contained in the respective GAs with PRODUVEN and FONDOIN (in bold italics above) will be used in this Review because they are closely aligned with the organizational delineation of responsibilities, and more accurately reflect the distinct functions of the two entities implementing the project . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Component 1: CFC Production Phase -Out (Cost at Appraisal: US $16.05 million; Actual Cost: US $16.47 millon). This project component was intended to ensure that Venezuela ’s CFC production at PRODUVEN ceased earlier than mandated by the MP through a compensation program . It included: (i) a compensation package to PRODUVEN for loss of profits and labor payroll for workers laid off due to CFC production closure (US$16.05 Million); and (ii) dismantling and retrofitting the PRODUVEN facility's equipment (reactors, storage, pumps gauges ) to produce only non-ODS CFCs and safely disposing of that equipment and existing stocks in compliance with the Equipment Disposal Plan. Component 2: Technical Assistance to FONDOIN (Cost at Appraisal: US $950,000 ; Actual Cost: US $420,000 ). . This component was designed to provide additional resources to help strengthen the national focal point for ODS in Venezuela; to create the institutional and legal mechanisms to monitor and enforce any renewal of ODS production; to control the import or export of illegal ODS; and to support the early replacement of ODS -using refrigeration chillers through the use of a revolving fund scheme . This last sub-component (Chiller Replacement Program, CRP) had an estimated cost of US$1 million, half of which would come from the grant and half from the GOV . It has not yet been implemented (see Sections 2d and 3 below). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Cost : Actual project cost ($16.47 million) was less than estimated ($17.0 million), mainly due to the MLF's decision Cost: not to approve funding (US $500,000) for the CRP. The reason for the decision cited in the ICR is the lack of a robust institutional underpinning for a revolving fund scheme (see Section 3 below) Project Financing : The project was financed entirely by the Montreal Protocol's Multilateral Fund (MLF). Of the US$950,000 grant destined for FONDOIN, US$530,000 was cancelled since the US $500,000 for the CRP was never approved or implemented, and the other US $ 30,000 was not required by FONDOIN to set up its enforcement mechanisms to control the import or export of ODS -causing CFC chemicals in Venezuela . Borrower Contribution : It was originally intended that the Borrower would contribute US $ 500,000 in matching funds for the CRP revolving fund, but this contribution was not made since the CRP was not implemented . Dates: Dates The closing date given by both draft Grant Agreements was June 30, 2009, but the Director's Memo and ICR both show the original (and actual) closing date as December 31, 2009. 3. Relevance of Objectives & Design: high . The Montreal Protocol on Substances that Relevance of Objectives : Relevance of the PDOs is rated as high. Deplete the Ozone Layer was adopted in 1987. It binds its signatories to actions limiting and eventually phasing out the use and production of ODS. Venezuela ratified the Montreal Protocol in 1989, and was eligible for financial support from the MLF to cover the incremental costs of meeting Montreal Protocol requirements . Thus, the project was consistent with Venezuela ’s obligations under the Montreal Protocol and with the Bank's CAS (2002) to help the country fully comply with the Montreal Protocol . No more recent CAS or CPS is available in the Bank's electronic database to compare the relevance of these objectives to a more current CAS or CPS . Relevance of Design : Design drew on the experience of previous ozone depletion projects in the Latin America Region. Output indicators related well to the PDOs . However, the Bank seriously underestimated the difficulties associated with the establishment of a revolving fund for the CRP in Venezuela . A credible system to ensure client tracking and repayment of loans could not be established . There were no local institutions accustomed to administering revolving loan funds . FONDOIN had no experience with running this type of program . No capacity building or other mitigating measures were built into the project . Consequently, the MLF did not release the US$500,000 portion of the Grant to FONDOIN corresponding to the funding of the CRP . The drafting of the PDOs in the Director’s Memo was, moreover, confusing, since it failed to attribute responsibilities clearly to the two different entities (PRODUVEN and FONDOIN) implementing the project. Design is rated as modest . 4. Achievement of Objectives (Efficacy): The first PDO of phasing out the production of ODS in Venezuela was fully achieved within the time frame established. All CFC-11 and CFC-12 production was eliminated by the end of 2006 and all of the production equipment was dismantled, retrofitted and disposed of in accordance with the Environmental Management Plan (EMP) and Equipment Disposal Plan. PRODUVEN adopted an M&E program, reporting quarterly to the Bank's supervision staff and subject to annual verification by an independent auditing firm . This was supervised by the Bank, FONDOIN and Ministry of the Environment and reported back to the MLF . The compensation package to PRODUVEN for lost profits and lost wages to its workers was disbursed as planned . None of this would likely have happened as quickly, if at all, without the support and impetus provided by the project . Achievement of this PDO is rated as high. high Achievement of the second PDO implemented by FONDOIN, acting as the Government ’s National Ozone Unit, was substantial . The project supported the establishment of appropriate regulations and institutions to monitor and enforce the implementation of the ODS elimination program . Guidelines were developed to identify, recover and destroy ODS, and a public awareness campaign was launched . Although the CRP was not implemented, the positive impact of the legal and institutional controls over the import or export of ODS -causing CFCs far outweighed that of replacing the 300 chillers existing in Venezuela at the time of appraisal (which accounted for only 0.05% of Venezuela’s annual ODS output). The legal and institutional controls put in place by FONDOIN would likely not have occurred without the project. 5. Efficiency (not applicable to DPLs): The implementation of the CFC production closure objective was completed in a satisfactory and cost -effective manner with no delays. The project's cost effectiveness (CE) ratio at closure indicates an improvement on the appraisal estimate. The latter estimated that the project would reduce ODS by 3,369 metric tons (MT) using project funds of US $16.47 million, resulting in a cost per ton of US$ 4,897. However, the ICR reports that the actual reduction achieved with that expenditure was 4,703 MT, or US$ 3,508 per ton, 28% below the appraisal estimate. FONDOIN’s activities were not subjected to a CE analysis, but were completed within budget . Efficiency is rated as substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project fully achieved its objective of eliminating production of ODS -causing CFCs in a cost-effective manner with no delays and ahead of schedule . The equipment used was dismantled, refitted and disposed of following good environmental practices. Compensation was paid to PRODUVEN and its labor force in accordance with the requirements of Venezuelan law. Although the CRP was not implemented, all other objectives of the TA to FONDOIN were attained. The relevance of the project ’s objectives was high, though that of design was modest . Outcome is rated satisfactory . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: Risk to development outcome is rated as low, given that all production of ODS has been shifted over to non -ODS HCFC-22 by PRODUVEN and all production equipment has either been dismantled or retrofitted for other non -ODS production. There is also adequate government control and monitoring of the import or export of ODS -causing CFCs with independent confirmation and verification . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: at -Entry : The Bank provided valuable support to the Government in its efforts to Bank Performance Quality -at- engage the Executive Committee of the Montreal Protocol in a dialogue to eliminate ODS -causing CFC production in Venezuela. The guidance and technical assistance was consistent with the Bank's country assistance strategy by helping Venezuela comply with its obligations as a signatory to the Montreal Protocols . Performance, nonetheless, had significant shortcomings . As noted in Section 3 above, the institutional and regulatory preconditions for the CRP financed by a revolving fund were inadequately researched and later found to be lacking. Moreover, the change in structuring the project ’s objectives in the two Grant Agreements and the Director’s Memorandum was both unnecessary and inappropriate; it was also a cause of confusion in the text of the ICR. QAE is rated moderately unsatisfactory . Bank Performance Quality of Supervision : The ICR reports that periodic supervision missions and other reporting mechanisms ensured good compliance with Bank safeguard, procurement and other fiduciary requirements, as well as satisfactory progress against output targets . The project was completed on-time and within budget, and the risk-to-development outcome is considered negligible . Supervision is rated satisfactory . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Borrower : The Borrower's (i.e., FONDOIN’s) performances is rated as satisfactory . FONDOIN achieved all its sub-objectives, such as strengthening the technical capacity of customs officers and other import /export control staff to identify and prohibit illicit production or importation of ODS -causing chemicals once production at PRODUVEN was eliminated. It successfully designed and implemented guidelines, standards, and procedures for recovering and destroying ODS . It created a monitoring and reporting system to track the Phase -out Plan’s execution, and verified the phase -out of ODS-causing CFCs by PRODUVEN in a transparent and credible manner. Implementing Agency : The implementing agency's (i.e., PRODUVEN’s) performance is satisfactory . PRODUVEN ceased production of CFC completely before the end of 2006, ahead of schedule, thereby allowing Venezuela to fulfill its commitments to the Montreal Protocol . It also dismantled, retrofitted, and disposed of all equipment and unused materials contaminated by CFC -11 and CFC-12 according to its Environmental Management Plan and Equipment Disposal Plan . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : The two output indicators (phase out of production of CFC -11 and CFC-12 as well as the dismantling, retrofitting and disposal of plant equipment needed to produce those ODS ) were measurable and well linked to the first PDO. The three additional output indicators (production caps and bans on ODS imports, technical capacity to monitor and intercede illegal imports or exports of CFC developed, and public awareness campaign conducted) were appropriate for monitoring the second PDO . M&E Implementation : To monitor and enforce the CFC production phase out plan, PRODUVEN was required to adopt an M&E plan and report quarterly CFC production figures to FONDOIN and the Bank . PRODUVEN’s plant was also subjected to annual verification by an independent auditor supervised by the Ministry of Environment (MOE). For its part, FONDOIN submitted semi-annual progress reports on performance to the Bank and MOE . These reports and audits were also made available to the MLF in line with their implementation follow -up responsibilities. s. M&E Utilization : This was difficult to ascertain since there is no evidence that this M&E information changed any management decisions, allocation of resources, or the direction of the project's implementation . a. M&E Quality Rating : Substantial 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : The project was classified as Category C for the purposes of OP 4.01. The ICR reports that an Environmental Assessment was, nonetheless, carried out and an Environmental Management Plan (EMP) was developed to ensure careful management and thorough purging of the ODS feed stocks into the plant reactors . The latter were dismantled, retrofitted, and disposed of in accordance with the Equipment Disposal Plan (EDP), which, in turn, reflected good industry practices . Fiduciary Responsibilities : The ICR reports that procurement and financial management were carried out in compliance with the requirements specified in the GAs and according to the relevant Bank stipulations . Financial audit reports were submitted on-time and followed all applicable Bank procedures satisfactorily . The ICR reports no qualifications of auditors’ opinion 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Moderately Quality-at-entry presented significant Satisfactory shortcomings (see Section 8 above) Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Whenever an innovative or locally unknown funding mechanism is planned (in this case, the revolving loan fund to retire ODS chillers early through interest -free loans), it is important to conduct feasibility and institutional analyses of local implementers to identify weaknesses or gaps in their capabilities and incorporate capacity-building measures into the design of the project or program . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is thorough and contains all the information and analysis necessary to evaluate the project . One weakness is the rather confused discussion in paragraph 34 regarding whether funding for the CRP had been approved or not by the Executive Committee of the MLF . It would have been interesting to compare this project ’s cost effectiveness with that of similar operations in the Latin American Region . The ICR’s presentation would have benefited from editorial revision by a native English speaker . a.Quality of ICR Rating : Satisfactory