ICRR 13318 Report Number : ICRR13318 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 05/25/2010 PROJ ID : P076798 Appraisal Actual Project Name : Kn: Hiv/aids US$M ): Project Costs (US$M): 4.5 4.6 Prevention And Control Project Country : St. Kitts & Nevis Loan/ US$M): Loan /Credit (US$M): 4.05 3.37 Sector Board : HE Cofinancing (US$M): US$M ): 0 0 Sector (s): Health (72%) Central government administration (23%) Law and justice (5%) Theme (s): Population and reproductive health (25% - P) HIV/AIDS (25% - P) Personal and property rights (24% - P) Health system performance (13% - S) Child health (13% - S) L/C Number : L7156 Board Approval Date : 01/22/2003 Partners involved : Closing Date : 06/30/2008 06/30/2009 Evaluator : Panel Reviewer : Group Manager : Group : Judith Hahn Gaubatz Susan A. Stout IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: The original objectives , according to the Project Appraisal Document (PAD), were to assist the Borrower to control the spread of HIV/AIDS through the following specific objectives : (i) scaling up prevention programs targeting high risk groups and the general population; (ii) strengthening treatment, care and support for people living with HIV /AIDS (PLWHAs); and (iii) strengthening St. Kitts and Nevis' multisectoral institutional capacity to better respond to the epidemic. According to the Loan Agreement (LA), the overall objective was also to control the spread of HIV /AIDS, though the specific objectives were articulated slightly differently as follows : (i) scaling up programs for prevention, care and control of the epidemic targeted in particular to high -risk groups; (ii) heightening awareness with respect to infection and prevention amongst the population; and (iii) strengthening the institutional capacity of the Ministry of Health (MOH), other related government agencies and civil society organizations to ensure the effectiveness and sustainability of the project. However, this review is based on the PAD objectives which are more specific than the LA objectives. The project was formally restructured, with the following revised objectives : to assist the Borrower to control the spread of HIV/AIDS and to mitigate its impact through the following specific objectives : (i) scaling up prevention services for high risk and vulnerable groups and the general population; (ii) expanding and strengthening treatment, care and support for PLWHAs and mitigating its impact on infected and affected persons; and (iii) strengthening the institutional capacity of the MOH, other government agencies, and civil society organizations (CSOs) to ensure an effective multi-sectoral response to the epidemic . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 05/09/2007 c. Components (or Key Conditions in the case of DPLs, as appropriate): Original components were as follows: (1) Advocacy and Behavior Change (Appraisal: US$0.5 million). Activities included creating a favorable legal and policy environment to project PLWHA rights, maintaining awareness levels of HIV /AIDS, and promoting behavior changes for risk reduction . (2) Prevention in High-Risk Groups and General Population (Appraisal: US$1.1 million). Activities included prevention and behavior change interventions with high risk groups (commercial sex workers (CSWs), men having sex with men (MSMs), uniformed personnel, migrant workers, and tourism workers ), as well as youth. Interventions included information and education campaigns (IEC), voluntary counseling and testing (VCT) services, condom distribution, prevention of mother -to-child-transmission ( PMTCT), and sexually transmitted infections (STI) management. (3) Improving Access to Treatment, Care and Support for PLWHA s (Appraisal: US$1.9 million). Activities included strengthening management of opportunistic infections (OI) and palliative care, antiretroviral treatment (ARV), health facilities and laboratory capacity, and home - and community-based care. (4) Surveillance, Epidemiology, and Research (Appraisal: US$0.5 million). Activities included implementation of an integrated management information system (MIS), second generation surveillance system, and training on project monitoring and evaluation. (5) Sustainable Management Response (Appraisal: US$0.3 million). Activities funded the creation and initial operations of the "National Advisory Council on HIV/AIDS" (NACHA), which would report to the Office of the Prime Minister. Its intended role would be to oversee resource mobilization for and implementation of HIV /AIDS activities. Revised components were as follows: (1) Scaling up HIV/AIDS Response by Civil Society Organizations and Line Ministries (Revised: US$0.2 million; Actual:US$0.13 million ). Activities included scaling up ongoing activities such as IEC, behavior change communications (BCC), sensitization and advocacy to address stigma and discrimination, condom distribution, and care for PLWHAs and orphans. Selected line ministries, including Education, Youth, Social and Community Development, and Gender Affairs were to provide services for both internal and external clients . (2) Strengthening the Health Sector Response (Revised: US$3.0 million; Actual: US$3.0 million ). Activities included strengthening VCT, condom distribution, STI management, laboratory services and blood safety, PMTCT, OI management, and ARV. The component also financed upgrading of physical facilities . (3) Strengthening Surveillance, Monitoring and Evaluation, and Institutional Capacity for Program Management (Revised: US$0.8 million; Actual: US$0.4 million ). Activities included development of a national M&E framework, epidemiological surveillance, and an information system . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project was formally restructured in May 2007 (at which point only US$0.45 million had been disbursed for activities including VCT services, purchase of condom vending machines, and purchase of equipment for health centers), due to slow implementation. The project closing date was extended for one year to allow for completion of activities. Almost 70% of the actual project funds were spent on strengthening health facility and lab capacity, while only 6% was spent on prevention interventions . Original counterpart funding was estimated at US$ 0.46 million; however, construction costs increased and the government, although initially agreeing to increase its contribution to $2.1 million, ultimately contributed $1.05 million, twice the appraisal amount. 3. Relevance of Objectives & Design: The relevance of the original objectives is rated Substantial, with caveats . Although there were no estimates available of the HIV prevalence rate in St . Kitts and Nevis specifically, there were estimates of other countries in the Caribbean (ranging from 1.2% in Jamaica to 3.2% in Trinidad and Tobago), and the region overall was deemed to be "at risk of seeing the epidemic 'generalized' (i.e. spread from high risk groups to the general population with an overall rate of about five percent ), if action was not taken. The project objectives are consistent with Bank and regional (Organization of Eastern Caribbean States (OECS)) development strategies which, while not specifically prioritizing the health sector, identifies "reducing vulnerability" as a main pillar. However, objectives as stated are primarily focused on process rather than actual outcomes . Given the short time frame of the project, it was appropriate to not focus on prevalence rates as the primary measure of impact . More intermediate outcomes, such as change in risky behavior or quality of life for PLWHAs, could have been included as objectives. The relevance of the revised objectives is rated Substantial. As noted above, the revised objectives are consistent with Bank and regional development strategies, although still not based on an adequate assessment of the HIV /AIDS situation in the country. Increased emphasis appears to be given to the health sector, which plays a critical role in the response to HIV/AIDS. However, the justification for additional emphasis on “mitigation of impact� may be questioned, as the cumulative number of PWLHAs (which the ICR reports as 301) represents less than one percent of the total population (49,000). The relevance of the original design is rated Modest. Although the range of project activities was very broad, they were likely to contribute to achieving the objectives within the time frame of the project, if implemented effectively . However, institutional arrangements were complex given the limited experience and capacity of the Borrower, in addition to the added dimension of working with a twin island federation . The design supports creation of a multisectoral institutional body (NACHA), although the PAD describes its role in terms of an eventual rather than an actual role, thus muddling the institutional responsibilities between NACHA and MOH . In addition, some elements of the design were not adequately assessed, including non -health sector engagement level , CSO engagement level and capacity, and M&E implementation arrangements . The relevance of the revised design is rated Substantial, with caveats . The revised design simplified implementation arrangements in the areas of procurement, financial management, and grant application procedures . The BSS was conducted in 2005-06, but it is not entirely clear that the data generated from the survey affected the design of interventions. It is also unclear from the ICR how the revisions of institutional arrangements affected the responsibilities of MOH vs. NACHA., given the increased emphasis on the health sector . 4. Achievement of Objectives (Efficacy): Original Objective 1: To Scale -Up Prevention Services Targeting High -Risk Groups and the General Population Achievement of this objective is rated Modest . Targets were achieved in some areas with regards to increasing the numbers of beneficiaries. However, there is limited information provided on levels of coverage or on the impact on behaviors (i.e. condom use, number of sexual partners ) that could ultimately affect prevalence . Outputs: At least 15 IEC campaigns and projects on HIV /AIDS prevention were carried out between 2005 and 2008. 45 condom vending machines were purchased, and 158,080 condoms were distributed (achieving the target of 150,000 condoms distributed). 1,564 persons in "most-at-risk" population groups were reached by prevention activities (including 1,051 MSM and 439 CSWs), however, there is no information provided on levels of coverage . Outcomes: 85% of men aged 25-49 years old reported using a condom at the last sexual encounter with a sex worker . The ICR notes that this figure, drawn from the 2005/06 OECS BSS, should be interpreted with caution due to the low denominator. In addition, while individuals in these groups may have been reached by the project's IEC and condom distribution activities, the ICR reports that these figures are based on the work of an NGO that did not r eceive direct project funding . The ICR also reports that stigma and discrimination constrained the project team from doing much work that directly targeted high risk groups . The ICR cites implementers interviewed who indicated that more people and communities (general population) were aware of HIV/AIDS today than before the project started . There is no quantitative evidence presented on increased knowledge, changed behavior, or impacts on other STDs . The number of pregnant women who received information about PMTCT services did not increase significantly (from 435 women in 2006 to 440 women in 2009). There is no data available on the number of HIV -positive pregnant women who actually received a complete course of ARV . Original Objective 2: To Strengthen Treatment, Care and Support for PLWHAs Achievement of this objective is rated Modest . Data provided in the ICR show an increase in the number of persons receiving treatment, care and support services; however, there is no information on whether those services were "strengthened" (i.e. improved in quality or effectiveness ). Outputs: 1,558 individuals received VCT services (surpassing the target of 770 individuals). ARV drugs and treatment were provided to 36 PLWHAs (falling short of the target of 43 PLWHAs). 15 PLWHAs received regular food vouchers from December 2008 to June 2009 for nutritional support to improve adherence to treatment programs (falling short of the target of 20 persons). Three health centers and one medical facility were constructed and /or rehabilitated and equipped. Outcomes: The number of PLWHAs receiving services increased; however, there is no information provided on levels of coverage, nor the effectiveness of those services . However, the ICR reports that the Global Fund provided funding for treatment, making it difficult to determine attribution of treatment outcomes to the project itself . Original Objective 3: To Strengthen Multisectoral Institutional Capacity to Better Respond to the Epidemic . Achievement of this objective is rated Modest . While various implementing partners were able to plan and carry out HIV/AIDS activities, there is limited evidence of "strengthened" capacity (i.e. improved skills or quality of services provided) or increased effectiveness of the Country ’s response to HIV/AIDS. Outputs: The National Council on HIV/AIDS was established with some project staff; however, there is little information provided in the ICR on the functions and effectiveness of this entity, particularly as a multisectoral institution . A six-country (OECS) behavioral surveillance survey was conducted in 2005-06. Five line ministries (including Gender, Education, Tourism, and Labor planned and implemented HIV /AIDS activities within their sectors. Four of the line ministries were also able to submit monitoring forms based on their activities. The ICR reports that approximately 20 percent of project staff participated in capacity building training, though it does not specify the nature or extent of the training activities . However, a critical project activity - the establishment of the management information system - was not completed to enable monitoring of the response . Outcomes: The number of non-health sectors implementing HIV/AIDS activities increased, as no line ministries were implemented HIV/AIDS activities prior to the project; however, there is limited information provided on an increase in effectiveness of the multisectoral approach, particularly as the ICR reports that multisectoral activities were not initiated until 2007. In addition, the ICR reports that HIV /AIDS activities have not been mainstreamed into work plans and budgets . The ICR cites strengthened institutional capacity as a contributing factor to successfully leveraging financing from other donors such as the Global Fund . Revised Objective 1: To Scale -Up Prevention Services for High -Risk and Vulnerable Groups and the General Population Achievement is rated Modest, based on the evidence cited above for original objective 1. Note: Although the revised objective adds “vulnerable groups� , it is the judgment of this review that “vulnerable groups� such as women, children and youth, are also part of the “general population� and therefore outcomes for this sub -group are not evaluated separately from the general population . Revised Objective 2: To Expand and Strengthen Treatment, Care and Support for PLWHAs and to Mitigate the Impact of HIV /AIDS on Infected and Affected Persons Achievement is rated Modest, based on the evidence cited above for original objective 2 and the lack of evidence on the added sub-objective to “mitigate impact�. Outputs: The ICR cites sub-projects addressing stigma and discrimination issues; however there is no specific information provided the activities that were carried out . Outcomes: According to the OECS BSS, only 1-5% of the population aged 15-49 years old expressed accepting attitudes towards PLWHAs. Revised Objective 3: To Strengthen Institutional Capacity of MOH, other line agencies and CSOs to Ensure an Effective Multi -Sectoral Response Achievement is rated Modest, based on the evidence cited above for original objective 3 and the lack of information on the implementation of activities to strengthen MOH specifically . Outputs: Three staff members of MOH participated in a four -month monitoring and evaluation internship in a neighboring country. In addition to the line ministries cited previously, 13 CSOs and 5 faith-based organizations (FBOs) – planned and implemented HIV/AIDS activities. Of these, 6 CSOs, and 3 FBOs were also able to submit monitoring forms based on their activities. Outcomes: (no additional outcomes cited) 5. Efficiency (not applicable to DPLs): Efficiency of the original project is rated Modest, due to the lack of information on the efficiency of project activities. The ICR cites literature which provides general data on comparative cost -effectiveness of various prevention interventions, a number of which were implemented in this project - however, there is no information provided on these interventions in the specific context of this project . Other project elements indicate the low efficiency of the project, including the high cost per person treated, the high cost inflation for one of the medical facilities, and the lack of implementation of activities targeted to high -risk groups, which would have been one of the most cost-effective means of prevention . Efficiency of the revised project is rated Modest, due to the same reasons cited above . No additional information on efficiency is presented for the revised project activities . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: According to ICR guidelines, the outcome rating of a restructured project is based on a weighted average (calculated by percent of the loan disbursed before and after the date of restructuring ). US$0.45 million, or 11.1% of the loan was disbursed before the restructuring . The outcome of the original project is rated Moderately Unsatisfactory, due to substantial relevance but modest achievements in prevention, care, treatment and capacity building, as well as lack of information on efficiency . The outcome of the revised project is also rated Moderately Unsatisfactory, due to continued relevance, but modest achievements in prevention, care, mitigation, treatment, and capacity building, as well as lack of information on efficiency . The overall project outcome is rated Moderately Unsatisfactory. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Risk to Development Outcome is rated Moderate. Capacity that was built through the project may be sustained as all project staff were absorbed into the government agencies . The ICR reports that the government “ is committed to provide free ART� after the Bank and Global Fund projects end, although there are no specific details of that commitment provided. It is not clear whether the government will also continue to provide grants to the line ministries and CSOs for prevention activities . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: at -entry is rated Moderately Unsatisfactory . The project emphasized key activities that were likely to Quality -at- lead to the intended outcomes, such as targeting high risk groups, and aimed to remove obstacles for a more effective response, such as creating a more favorable environment for PLWHAs and improving surveillance . However, the project was overly complex given the limited institutional capacity and engagement of non -MOH partners. Project design features were not based on a clear understanding or knowledge of the context of St . Kitts and Nevis (population: 49,000), such as procurement procedures which were not appropriate for a small island nation with limited suppliers or the context of small population . There was no baseline information on overall HIV prevalence levels nor any related indicators such as levels of risky behavior, prevalence levels in specific population groups such as youth or CSWs, or STI prevalence . Supervision is rated Moderately Satisfactory . The Bank team worked to restructure the project and resolve implementation problems, which led to the eventual completion of numerous activities and disbursement of the majority of the credit. However, there were four different task team leaders (with different task team members) after project effectiveness, which had a disruptive effect on implementation . The ICR reports the counterpart project team statement that each new team brought a learning curve that led to implementation delays and setbacks. The Bank team made concerted efforts to resolve M&E issues, including providing technical assistance to build capacity, harmonizing reporting requirements, and supporting second generation surveillance . However, a functioning M&E system that could provide essential data on the state of the HIV epidemic and the effectiveness of interventions was not established, although development was finally underway at the time of project closure. at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government performance is rated Moderately Satisfactory . The Prime Minister was a strong advocate in the fight against HIV/AIDS and a recognized leader in the region, and the ICR reports that the inclusion of HIV /AIDS on the government's agenda contributed to a more open environment for addressing the issue . The ICR reports that there were key staffing posts that were left unfilled for long periods of time, leading to serious implementation delays. Implementing agency performance is rated Moderately Satisfactory . A number of project activities were implemented, and procurement and fiduciary functions were adequately carried out, albeit with significant delays . All PMU staff were moved to regular government positions after the project closed, thus retaining the capacity built. The project team confirms that the heads of the Ministry of Health and the Ministry of Sustainable Development worked closely together to implement the project and provide leadership . However, several key activities related to achieving outcomes were not implemented . Prevention activities for high risk groups were not carried out as planned due to similar work being done by the Caribbean HIV /AIDS Alliance (which, according to the project team, was not receiving project funds but was exchanging information with the team ). The information system to support M&E was not implemented under this project as, according to the project team, technical assistance and financing for the information system were received from other donors . There is very little information on the effectiveness of the multisectoral institution, NACHA, created by the project . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design is rated Negligible. The design did not properly take into account the M&E capacity limitations of the country. The list of indicators to be tracked was lengthy, and there was no means for data collection, as surveillance systems and research capacity were inadequate . The revised set of indicators was in line with internationally-accepted standards; however, they were not adequately geared towards development outcomes . M&E Implementation is rated Modest. A six-country behavioral surveillance survey (BSS) was conducted in 2005-06 and provided some baseline data . However, a critical project activity - the setting up of the MIS was not completed . The ICR reports that this was due to the "varied, inconsistent, sophisticated, and expensive " nature of the technical advice given by Bank consultants that was not feasible given the country capacity and needs . An M&E framework, harmonized with the various donors, was eventually established, although the system itself was not fully developed and M&E reporting requirements were not fulfilled . A Knowledge, Attitudes, Practices and Behavior Survey is planned for 2010. M&E Utilization is rated Negligible. The ICR reports that although reports are shared with donors, “regular, systematic reporting to the Cabinet does not occur .� a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Procurement: The project encountered numerous procurement difficulties as the counterpart staff was not familiar with Bank procedures. The first procurement plan took over a year to develop . The ICR notes that the Bank's procurement guidelines did not fit in the context of a small island system with limited suppliers . After restructuring, in which procurement for civil works was simplified, the rate or procuring goods and services increased. Financial Management: The counterpart team was initially operating on outdated policies related to the management of special accounts, subsequently financial arrangements were modified (and simplified). The ICR reports that by the project closing, the financial management system was operating effectively, although monthly bank statements and audits were submitted late . Environmental: The project was assessed as environmental category "B" due to medical waste issues. In 2006, a biomedical waste management system was set up by a consultant, who plans to train staff on using the system. A medical waste autoclave system was recommended for the system but was not procured . Other positive impacts: The ICR reports that interviews conducted with project staff and beneficiaries indicated that the project had made a "significant difference in the lives of people from St . Kitts and Nevis, providing an environment where HIV and AIDS could be discussed ." However, the ICR also states, and the project team confirms, that stigma is still a significant barrier to a more effective response . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The outcome of the original project is Satisfactory Unsatisfactory rated Moderately Unsatisfactory, due to substantial relevance but modest achievements in prevention, care, treatment and capacity building, as well as lack of information on efficiency . The outcome of the revised project is also rated Moderately Unsatisfactory, due to continued relevance, but modest achievements in prevention, care, mitigation, treatment, and capacity building, as well as lack of information on efficiency. According to ICR guidelines, the outcome rating of a restructured project is based on a weighted average (calculated by percent of the loan disbursed before and after the date of restructuring ). US$0.45 million, or 11.1% of the loan was disbursed before the restructuring . Therefore, the overall project outcome is rated Moderately Unsatisfactory . Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately While Supervision was Moderately Unsatisfactory Unsatisfactory Satisfactory, Quality-at-entry was Moderately Unsatisfactory due to the complex design and inadequate M&E arrangements. According to the harmonized criteria, when one element of Bank performance is in the satisfactory range and the other in the unsatisfactory range, the project's outcome rating determines the overall Bank performance rating. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Project design should be specific to the country context, based on sufficient understanding and knowledge of the baseline situation and the country capacity . Although political commitment may exist at the highest level of government, the project design still needs to take into account the level of engagement of lower level implementers and include measures to increase commitment if needed. Frequent turnover of Bank staff can exacerbate difficulties of inexperienced counterpart staff in implementing Bank projects. 14. Assessment Recommended? Yes No Why? To verify the ratings and document lessons learned . As of yet, none of the Caribbean HIV /AIDS MAP projects has been the subject of a PPAR . This project could be considered as part of a clustered PPAR, especially in light of lessons that could drawn from supporting an HIV /AIDS response in a small country context . 15. Comments on Quality of ICR: The ICR is overall consistent with guidelines and presents a frank discussion of the project's achievements and shortcomings. However, the discussion of project outcomes was largely centered around the specific project activities. The ICR also lacked a thorough discussion of some institutional issues, in particular how the NACHA and MOH executed their roles, and how and why those roles changed as a result of the project restructuring . Annex 1 on project costs appears to be inaccurate, as the appraised project cost is shown as $ 8.24 million, compared to $4.5 million in the PAD. a.Quality of ICR Rating : Satisfactory