Document of The World Bank FOR OFFICIAL USE ONLY Report No: IMPLEMENTATION COMPLETION AND RESULTS REPORT TF 019188 ON A SMALL GRANT IN THE AMOUNT OF USD 2,750,000 MILLION TO THE The Carter Center FOR Supporting Psychosocial Health and Resilience in Liberia (P146591) April 2019 Health, Nutrition & Population Global Practice Africa Region Regional Vice President: Hafez M. H. Ghanem Country Director: Henry G. R. Kerali Senior Global Practice Director: Timothy Grant Evans Practice Manager: Gaston Sorgho Task Team Leader(s): Preeti Kudesia ICR Main Contributor: Munirat Iyaboe Ayoka Ogunlayi ABBREVIATIONS AND ACRONYMS CAMCH Child Adolescent Mental Health Clinicians CCC Community Care Centers CHT County Health Team CPS Country Partnership Strategy DALY Disability Adjusted Life Years DL Disbursement Letter ENACT Enhancing Assessment of Common Therapeutic Factors ETU Ebola Treatment Units EVD Ebola Virus Disease FCV Fragile, Conflict and Violent GBD Global Burden of Diseases gCHW General Community Health Worker GDP Gross Domestic Product GNI Gross National Income GoL Government of Liberia HDI Human Development Index JSDF Japanese Social Development Fund M&E Monitoring and Evaluation MBI Maslach Burnout Inventory MHC Mental Health Clinician MoE Ministry of Education MoH Ministry of Health MSPSS Multidimensional Scale of Perceived Social Support NCD Non-communicable Disease NDS National Drug Services NHMP National Mental Health Policy OIC Officer in Charge PAC Project Advisory Committee PDO Project Development Objective PHQ Patient Health Questionnaire PTSD Post-Traumatic Stress Disorder PW Psychosocial Workers RF Results Framework SBC School Based Clinic SOE Statements of Expenditure SSA Sub-Saharan Africa SW Social Worker SW Social Workers TCC The Carter Center TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME .................................................................................................................... 10 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 19 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 21 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 23 SUSTAINABILITY .................................................................................................................. 24 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 27 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 38 ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 39 ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 40 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P146591 Supporting Psychosocial Health and Resilience in Liberia Country Financing Instrument Liberia Investment Project Financing Original EA Category Revised EA Category Not Required (C) Organizations Borrower Implementing Agency The Carter Center The Carter Center Project Development Objective (PDO) Original PDO The Project Development Objective (PDO) is to respond to the intermediate psychosocial and mental health impact of the Ebola Virus Disease (EVD) crisis and to build long-term psychosocial health and resilience at the individual and community levels in project target areas. FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-19188 2,750,000 2,750,000 2,750,000 Total 2,750,000 2,750,000 2,750,000 Total Project Cost 2,750,000 2,750,000 2,750,000 Page 1 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) KEY DATES Approval Effectiveness Original Closing Actual Closing 25-Jan-2015 25-Feb-2015 25-Feb-2018 31-Dec-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 17-Aug-2017 2.15 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Financing Plan Reallocation between Disbursement Categories Change in Disbursements Arrangements Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 02-Oct-2015 Moderately Satisfactory Moderately Satisfactory 0.00 02 29-Jun-2016 Satisfactory Satisfactory 0.29 03 15-May-2017 Satisfactory Satisfactory 1.70 04 29-Nov-2017 Satisfactory Satisfactory 2.15 05 28-May-2018 Satisfactory Satisfactory 2.26 06 08-May-2019 Satisfactory Satisfactory 2.75 Page 2 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) ADM STAFF Role At Approval At ICR Regional Vice President: Makhtar Diop Hafez M. H. Ghanem Country Director: Yusupha B. Crookes Henry G. R. Kerali Senior Global Practice Director: Timothy Grant Evans Timothy Grant Evans Practice Manager: Trina S. Haque Gaston Sorgho Task Team Leader(s): Rianna L. Mohammed-Roberts Preeti Kudesia Munirat Iyabode Ayoka ICR Contributing Author: Ogunlayi Page 3 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES 1. Liberia, a post-conflict fragile state with abundant natural resources, but one of the poorest countries in the world with a rapidly growing population estimated at 4.12 million, with 63 percent of the population below age of 25 years. Despite robust annual growth of 8.3 percent in 2012, the gross national income (GNI) per capita is only US$370, far below the average for other low-income countries in Sub-Saharan Africa (SSA). More than one- half (56 percent) of the population lives below the national poverty line and approximately 80 percent lives below the international poverty line of US$1.25 per day.1 Despite gains in recent years, improving human development outcomes in Liberia continues to be challenging. In 2013, Liberia ranked 174th out of 186 countries on the United Nations Development Program Human Development Index (HDI) with a score of 0.388. The score was significantly lower than the SSA average.2 Despite its 170 years’ history, Liberia is still building its national identity as still large segments of Liberians having greater allegiance to their tribe (there are 16 tribes in addition to Americo-Liberians and Congos) than to Liberia. 2. The civil unrest and instability due to two devastating civil wars which lasted from 1989 – 1997 and 2001 – 2003, left Liberia with a weakened and seriously dilapidated health system incapable of adequately responding to the health needs of the populace. Even though Liberia made notable progress in health systems management and health services delivery over the last decade, the country continued to face significant challenges in further improving health outcomes. Available research showed that mental health and psychosocial issues affected a large portion of the population in Liberia and largely stem from the civil war during which a significant portion of the population experienced various traumatic events. A 2008 study, for example, found that 40 percent of the Liberian population self-reported symptoms characteristic of major depression, and 44 percent noted indications associated with post-traumatic stress disorder (PTSD).3 The 2010 Global Burden of Disease (GBD) also revealed that mental disorders in Liberia accounted for more disability-adjusted life years (DALYs) than any other non- communicable disease (NCD).4 Scientific evidence found linkages between these and other mental disorders and reduced functioning and engagement in high-risk behaviors (e.g., substance abuse and interpersonal violence). 3. The above situation was further exacerbated by the Ebola Viral Disease (EVD) outbreak in 2014 with over 4,000 cases confirmed and over 3000 deaths reported. The crisis significantly overstretched an already weak health system and impacted in all sectors of the economy, and reduced Liberia’s gross domestic product (GDP) growth from 5.9 percent growth to 2.5 in 2014.5 Field reports confirmed the toll of the EVD outbreak on community and individual psychosocial well-being. The observed high levels of stress as well as individual and community grief reactions were a result of the fear and major loss which occurred due to EVD outbreak. Together, the heightened fear and suspicion, the loss of family and friends, as well as the inability to mourn the deceased according to traditional and religious rituals, was reported to cause the population further stress and the potential to 1 World Bank. 2013. Liberia Overview. Available online: http://www.worldbank.org/en/country/liberia/overview 2 UNDP. 2013. Human Development Report. Available online: http://www.undp.org/content/dam/undp/library/corporate/HDR/2013GlobalHDR/English/HDR2013%20Report%20English.pdf 3 Johnson K, Asher J, Rosbourough S, et al. 2008. Association of combatant status and sexual violence with health and mental health outcomes in post- conflict Liberia. JAMA. ;300(6):676-90 4 In addition, neuropsychiatric disorders contribute to an estimated 4.7 percent of the global burden of disease (WHO 2011). The 2010 Global Burden of Disease shows that major depressive disorder contributes to 2.87 percent of the total DALYs in the non-communicable disease (NCD) category among Liberia’s working age population. Global Burden of Disease 2010. 2013. Available online: http://viz.healthmetricsandevaluation.org/gbd-compare/ 5 World Bank. 2014 The Economic Impact of the 2014 Ebola Epidemic: Short and Medium Term Estimates for Guinea, Liberia, and Sierra Leone. Working Paper 90748. Page 4 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) exacerbate previously unaddressed poor psychosocial and mental health. On the individual level, research shows that unmitigated trauma in many instances leads to negative coping behaviors (e.g., substance use and poor self- care), a limited sense of agency (self-efficacy), and hopelessness about the future. On the community level, the trauma manifested in a lack of collective efficacy, a general mistrust of formal institutions, and a reduction in social cohesion. In addition to confronting suffering and death on a daily basis, staff at Ebola Treatment Units (ETUs) and Community Care Centers (CCCs), and burial teams (“first responders�) faced serious stigma in their own communities. Additionally, in the face of stress, some first responders had reportedly resorted to maladaptive coping behaviors including excessive alcohol use. 4. Left untreated, the situation could hinder individual’s ability to fully and productively participate in daily life which could have important consequences for a country such as Liberia which relies on the population for its short- and long-term recovery. In addition to the individual-level impact, available research pointed to the impact of traumatic events on both community and family ties. 5. Recognizing the importance of addressing the psychosocial and mental health needs of its population, the GoL developed a National Mental Health Policy (NHMP) in 2009, which prioritized an integrated and decentralized approach to the delivery of mental health care and focused on the following objectives: (i) prevention of mental illnesses; (ii) improved accessibility and availability of psychosocial and mental health services; (iii) provision of services for particularly vulnerable groups; (iv) provision of rehabilitative services and; and (v) the provision of social services. Achieving these objectives through a community-based approach was one of the pillars of the GoL’s strategy to address the mental health and psychosocial needs of the population. Rationale for Bank Involvement 6. Despite GoL’s efforts in developing the NMHP, operationalizing the policy was a challenge due to lack of resources to cover the costs associated with mental health prevention and treatment and insufficient number of trained mental health care workers to meet the significant demand of psychosocial and mental health care and services in the country. This project helped to address the gap to a large extent by making resources available to address the post conflict trauma and psychosocial issues that arose because of EVD. 7. This project strengthened protective factors as a means of promoting resilience. Individuals who are resilient are more likely to demonstrate an adaptive stress response, recover from stressful events, and be less susceptible to stress-related psychopathologies and poor mental health. The project thus contributed to addressing psychosocial health impact of the EVD outbreak at both the individual and community levels, whilst also building long-term resilience and psychosocial health. 8. Project’s design and objectives aligned well with: (1) GoL’s Psychosocial Response Strategy and Procedures Addressing the Needs of Persons, Families and Communities Affected by Ebola in Liberia, (2) National Mental Health Policy and the Operational Plan for Accelerated Response to Re-occurrence of Ebola Epidemic, and (3) World Bank Country Partnership Strategy (CPS) 2013 - 2017, which highlighted improvements in human development as one of the Bank’s strategic areas of focus in Liberia, with investment in the health sector noted as being critical for Liberia’s transformation from a fragile state to a middle-income country by 2030. 9. The project was a trust fund grant fully financed by the Japan Social Development Fund (JSDF). The JSDF grant requirements specified third party implementation for which the Carter Center (TCC) was designated as Page 5 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) implementing agency as neither the World Bank or the government of Liberia could be project implementing agency. The TCC has been present in Liberia for over 20 years in implementing mental health and had excellent working relationship with the Ministry of Health and Social Welfare in promoting long-term mental health. Project Development Objectives (PDOs) 10. The project development objective was to respond to the intermediate psychosocial and mental health impact of the EVD crisis and to build long-term psychosocial health and resilience at the individual and community levels in project target areas. Key Expected Outcomes and Outcome Indicators 11. The project paper (PP) stated three groups of beneficiaries of the project: (a) the individuals receiving (intermediate and longer-term) psychosocial support; (b) the larger community who participated in project activities; and (c) different cadres of Liberia’s mental health workforce (including Mental Health Clinicians (MHC), social workers (SWs), psychosocial counselors (PWs), gCHVs) who received training under the Project. Specifically, targeted beneficiaries under (a) and (b) included: (i) Ebola-affected individuals and their households; (ii) “first responders�; (iii) high-risk individuals and groups (i.e. mothers, children and adolescents (including unaccompanied children); and (iv) communities in the two targeted counties of Mosterado and Margibi. 12. The estimated number of direct project beneficiaries in the PP was 18,903. This total included: (i) individuals receiving or participating in psychosocial interventions (18,197); and (ii) individuals receiving training under the project (706). An estimated 300 communities (which generally range from 250 to 500 members) in the counties targeted by the project were expected to benefit from the project. 13. There were four PDO indicators to measure the achievement of the PDO in the original design of the project: • Increase by 55% the levels of competence, skills and confidence among providers; • Decrease by 20% depression, post-traumatic stress disorder (PTSD) and poor functioning among project beneficiaries; • Decrease by 30% stigma against Ebola-affected individual/households; and • Increase by 75% levels of trust at the community level. These indicators were revised at project restructuring (see details below). Components 14. The project had 3 components: Component 1: Support for Intermediate Mental Health/Psychosocial Needs (Allocated amount: US$0.51 million). This component responded to the intermediate psychosocial/mental health impact of the EVD crisis though the implementation of a set of psychosocial interventions aimed at addressing grief, trauma, and stigma at the individual and community levels. Component 2: Support to Build Long-Term Psychosocial Health and Resilience at the Individual and Community Level (allocated amount: US$1.99 million). The interventions under this component aimed at building resilience and long-term psychosocial health, by focusing on factors that promote and engender trust, efficacy, and acceptance of those with mental health issues at the individual and community levels. Psychosocial interventions Page 6 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) to enhance resiliency and promote health and well-being of vulnerable groups (including mothers, children and adolescents) were also supported under this Component. The results chain below listed the activities implemented in the above two components. Component 3: Project management, monitoring and evaluation, and knowledge dissemination (allocated amount: US$0.25 million). The activities under this component supported project monitoring and evaluation (M&E) as well as overall project management. 15. SIGNIFICANT CHANGES DURING IMPLEMENTATION 16. Revised PDOs - There was no change in the PDO during project implementation. Revised PDO Indicators and Outcome Targets 17. The project’s results framework (RF) was changed once. Two PDO indicators were revised, two were deleted and one new indicator was introduced. As for intermediary indicators, five indicators were revised, 2 new indicators were introduced, and no change made to four indicators. Also, the revision to the targets in the RF for some outcome and intermediate indicators was made because of new baseline data generated and implementation delays. In addition, two indicators, “Stigma against Ebola- affected individual/households� and “Levels of trust at the community level,� were deleted because they were tied to discrete project activities that Page 7 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) yielded only a one-time measure. They were replaced with two new indicators, one PDO indicator “Clinician stigma against individuals with mental illness� and one Intermediate Indicator “Levels of perceived social support�. Revisions made to the intermediate indicators are detailed in the Revised RF in annex 1. Table 1: Comparison of PDO Indicators in the Original and Restructured Project Paper (PP) Wording of Indicator Unit of measure End Target Original PP Restructured Paper PP Restructur PP Restructur ed Paper ed Paper PDO Indicator #1 Level of Level of competence, % % 55 66 competence, skills skills and confidence and confidence among providers. among providers (Note that the indicator is same, the only change was in the target for the indicator). PDO Indicator #2 Depression post- Improvement in % % 20 40 traumatic stress depression, post- disorder (PTSD) traumatic stress and disability disorder (PTSD), and among project disability among beneficiaries project beneficiaries. PDO #3 Clinician stigma % 50 (This is a new against individuals indicator with mental illness added) Deleted PDO Indicators Deleted PDO Indicators Reason(s) 1. Stigma against Ebola affected individual and households Indicator deleted because it was linked to a discrete project activity that yielded only a one-time measure. 2. Levels of trust at the community level Same reason as above Financing Change in Grant Closing Date 18. Formal project restructuring extended project's closing date by 10 months, from February 28, 2018 to December 31, 2018. This additional time helped to mitigate the delays in project implementation that stemmed from: (a) the Bank’s financial management (FM) team’s decision to change the required financial reporting method from Statements of Expenditure (SOE) to Interim Financial Report, due to the initial delays by TCC in making disbursement requests due to challenges in using SOE as stipulated in the original Disbursement Letter; Page 8 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) and (b) The inability to use project funding to finance CHT-incentives, as originally expected, and efforts to circumvent this constraint. Change in Disbursement Arrangements 19. The disbursement arrangements were changed prior to this restructuring. Specifically, the Disbursement Letter (DL) was revised to change supporting documentation from Statement of Expenditure to Interim Financial Report. The revised DL was dated April 15, 2016. Change in Allocations - Changes in funds allocation were based on the following justifications: 20. In September 2015, World Bank requested an exception from the Japanese Social Development Fund (JSDF) to allow TCC to cover salaries of TCC headquarters staff providing direct technical support to the JSDF Grant. Those salaries, which were budgeted under the Consultant Line in the agreed project costing based on prior Bank guidance, were later flagged as a conflict of interest by the Bank's procurement team as TCC was converting existing staff to consultants. In December 2015, the JSDF agreed to increase the operating cost threshold from 10% to 15% to help offset the salary expenditures of TCC staff providing direct technical support to the JSDF Grant. The proposed reallocation addressed the agreed budget adjustment. 21. In February 2016, the World Bank informed TCC—based on clear guidance from JSDF and the Bank’s fiduciary team—that staff incentives for County Health Teams (CHTs) could not be considered eligible expenses since CHT staff are civil servants, and hence cannot act/work as consultants for the project. To not delay project implementation further, TCC used its own non-JSDF funds to cover this funding gap. Project funding that was originally budgeted to support CHTs (including for staff incentives) was therefore available for reallocation. The table below highlights the reallocation of funds across the project’s four categories of expenditures. Original Current Category of Expenditure Reallocation Allocation 1 DISB - CONSULTING 458,258.00 182,005.00 2 DISB - TRAINING 1,989,742.00 2,154,300.00 3 DISB - OPERATING COSTS 250,000.00 358,695.00 4 DISB - GOODS 52,000.00 55,000.00 Total: 2,750,000.00 2,750,000.00 Change in Disbursement Estimates 22. The disbursement estimates were revised to reflect the slow start-up in the project implementation. This was largely due to lack of familiarity by the implementation agency in the use of Bank fiduciary procedures. Fiscal Year Original Disbursement Amount (USD) Changed Disbursement Amount (USD) Page 9 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) 2015 0.00 0.00 2016 800,000.00 237,000.00 2017 780,000.00 1,000,000.00 2018 780,000.00 1,327,000.00 2019 390,000.00 186,000.00 Total 2,750,000.00 2,750,000.00 Change in Implementation Schedule 23. The implementation schedule was revised due to implementation delays described above, and project implementation period extended by 10 months to December 31, 2018 in order to mitigate delays and ensure achievement of the PDO by project closing. II. OUTCOME 24. The project development objective was “to respond to the intermediate psychosocial and mental health impact of the EVD crisis and to build long-term psychosocial health and resilience at the individual and community levels in project target areas�. The PDO has 3 core indicators and 11 intermediate outcome indicators. Of the three core indicators, two indicators – i) Level of competence, skills and confidence among providers, and ii) Improvement in depression, post-traumatic stress disorder (PTSD), and disability among project beneficiaries, were achieved and surpassed targets while the remaining core indicator on “Clinician stigma against individuals with mental illness� was not attained even though the objectives seemed to be achieved. This was because of the way the indicators were worded. Nine of the intermediate outcome indicators: i) levels of stress and burnout among health workers (including first responders), ii) Levels of reported high risk health behaviors amongst responders, iii) Levels of perceived social support, iv) Communities receiving community level interventions, v) Individuals receiving direct support (peer groups and individual/group counselling), vi) Students receiving weekly support from CAMHCs, vii) Women’s health trainers whose trainees demonstrate improvement in their knowledge of women’s health, viii) Direct project beneficiaries (of which percentage female), and ix) Health personnel receiving training were achieved and surpassed targets; while the remaining two indicators – Completed referrals from general Community Health Workers (gCHVs) to Social Workers (SWs) to Mental Health Clinicians (MHCs), and Increased supervision and support of Mental Health Clinicians (MHCs) and Child and Adolescent Mental Health Clinicians (CAHMCs) were not achieved. The analyses of achievements and progress recorded are discussed by indicator below: Project Development Objective Indicators a) Level of competence, skills and confidence among providers The project trained and accredited a total of 102 Child Adolescent Medical Health Clinicians (CAMHC), and currently working in hospitals, clinics, and schools across Liberia to address specific issues related to child and adolescent mental health. The ENACT (Enhancing Assessment of Common Therapeutic factors) was used to evaluate the competency and practical skills of CAMHC cohorts trained. Averaged over five cohorts ENACT pre- test scores increased from an average of 29.8 to an average post-test score of 41, for an increase of 37.6 per Page 10 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) cent.6 The adjusted percentage improvement was even higher, moving from 32.9 per cent to 63.8 per cent, for a positive change of 93.9 per cent.7 While 15 per cent of CAMHCs had adequate clinical competency (as measured by a score of 37 or higher on the ENACT assessment) in their baseline or pre-training assessment, 67.7 per cent had adequate clinical competency at the post-training assessment. . This achievement surpassed target of 66 percent by 3 percent and above the baseline of 15 percent by 53 percent. b) Improvement in depression, post-traumatic stress disorder (PTSD), and disability among project beneficiaries The Patient Health Questionnaire (PHQ-9) was used to screen for depression. Cumulatively, of the 1,021 individuals with baseline scores, 100 (11.4%) had follow-up PHQ-9 scores. Of these 100 individuals, 66% (n=66) demonstrated improvement in symptoms of depression from baseline to follow up. This indicated that two-thirds of patients showed improvement after treatment. This surpassed target by 22 percent. In addition, a total of 29,114 were diagnosed and treated for various mental health conditions in health facilities in the 2 project targeted counties of Margibi and Monsterado during the project period from January 1, 2016 to December 30, 2018. Figure 1 below depict the numbers of mental cases diagnosed and managed during the life of the project. The highest number of cases managed was epilepsy and seizure (46.34%), followed by depression (19.96%), and psychosis (11.78%). Fig 1: Mental health conditions diagnosed and treated in health facilities in Margibi and Montserrado counties from January 1, 2016 to December 31, 2018 16,000 13,491 14,000 12,000 Number of cases 10,000 8,000 5,810 6,000 3,430 4,000 1,326 1,864 1,481 2,000 466 587 323 187 129 20 0 Mental Health Cases c) Clinician stigma against individuals with mental illness Working with health care providers on facility-wide anti-stigma activities was a vital aspect of the project, considering destructive features of mental illnesses with wide-ranging detrimental effects on patients, caregivers, 6 In cohorts 1-5, average pre-test scores were 32.5, 31.7, 30.3, 24.2, and 30.5, respectively. Post-test scores were 38.1, 45, 36.1, 47.4, and 38.2 for cohorts 1-5, respectively. Though the number of persons assessed in each cohort varied, the in evaluation weighted each cohort equally. 7 In cohorts 1-5, average adjusted pre-test percentages were 40.2 per cent, 38.0 per cent. 34.3 per cent, 17.3 per cent, and 34.6 per cent, respectively. Post-test adjusted percentages 55.8 per cent, 74.9 per cent, 50.4 per cent, 81.7 per cent, and 56 per cent for cohorts 1-5, respectively. Each cohort was weighted each quarter equally. Page 11 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) and communities due to stigma. Clinician stigma against persons with mental illness was measured using the Social Distance Scale. According to this metric, the percentage of individuals whose stigma level fell was 85.42% with a score of 11.11% in Year 1 and 79.59% with a score of 17.02% in Year 2. These scores represent different cohorts of individuals trained, rather than one group followed over time, although some of the individuals assessed in Year 2 were also assessed in Year 1. 64.29% of participants in Year 1 showed improvement in their scores from pre-test to post-test and overall there was a statistically significant mean difference in the Year 1 scores of -3.29 (Median: -2.5, SD: 6.15, p=.0013). For Year 2, 50% of individuals had an improved score from pre- to post-test, and there was a statistically significant mean difference from pre- to post- of -1.57 (Median: -.5, SD: 4.85, p=.0339). Even though this achievement was impressive, and in a way was achieved as a lower score indicating lower level of stigma is desirable, thus making the end target of 50% set inappropriate. The end target ought to have been adjusted given the baseline of 43.18% to set an end target that would be lower than baseline as an indication of progress in achieving reduction in stigma against mental illness. This observation was further confirmed from the project evaluation findings that “high rates of stigma is the norm within the health systems. Before the project it was common for health staff to refer to mental health patients as ‘crazy’ or ‘witched’; another common stigma was the belief that epilepsy is contagious. As indicated by one Mental Health Clinician (MHC) “before if you asked the health staff – even me – if epilepsy is contagious, everybody would say ‘yes’. Even in the training we were all shocked when we learned it wasn’t, and then learned the types of epilepsy and how to treat them�.8 Anecdotes like these were common amongst the health workers and signal the lasting transfer of knowledge and skills made to the health system through the trainings offered by the project. Intermediate Indicators d) Levels of stress and burnout among health workers (including first responders) The stress and burnout measured amongst health workers using the Maslach Burnout Inventory (MBI) include: emotional exhaustion (measures feelings of being emotionally overextended and exhausted by one's work), depersonalization (measures an unfeeling and impersonal response toward recipients of one's service, care treatment, or instruction), and personal accomplishment (measures feelings of competence and successful achievement in one's work). Cumulatively, of the 85 MHCs, SWs, mid-level health care workers and 369 first responders who completed MBI assessments, 86.71% on average demonstrated high levels of stress and burnout. Fifty-four percent (54.3%) had a high-risk score in one of the three domains, 22.8% had a high-risk score in two of the domains, and 9.61% had a high-risk score in all three domains. It is important to note that these are different respondents than those indicated in the baseline score (84.21%). Some of the first responders were engaged in the recovery groups and the program was able to follow their scores over time as they attended the groups. Even though this has surpassed end target, the method of measuring or assessing the indicator was not appropriately as it did not capture the impact of self-care and stress relief training for first responders as the training was for two days and assessment was conducted using pre-test at the beginning and post-test at the end of the training. This did not allow enough time to analyze the impact of the training once the responders started utilizing knowledge and skills acquired from the training. A qualitative assessment might have been more appropriate in demonstrating the impact of this intervention. 8 KII, Cotton Tree Health Center, 7 November 2018. Page 12 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) e) Levels of reported high risk health behaviors among responders: The CAGE screening tool was used to screen for alcoholism. A score of 2 or higher on a scale of 1 – 4 indicates risk for alcoholism. Cumulatively, of the 369 first responders who received self-care training during the project implementation, 107 completed CAGE assessments, and 59.8% (n=64) scored high risk on all CAGE assessments. f) Levels of perceived social support This indicator was measured using the Multidimensional Scale of Perceived Social Support. The baseline level of perceived social support was 71% (which exceeded initial end target of 60% and was adjusted to 78%). A follow up assessment conducted on the level of perceived social support reported 85% indicating an improvement. This achievement surpassed the adjusted target by 7% and 14% improvement from baseline of 71 percent. g) Completed referrals from gCHVs to SWs to MHCs Under the project, gCHVs were trained to live and work in communities attached to the mhGAP trained facilities and educate community members and families to support and include people living with mental illness. Including families was vital as the choice of mental health care often depends upon family’s perceptions of the cause of illness and choice of treatment. For instance, if the family felt witchcraft was the cause of mental illness, they were more likely to seek care from a traditional healer. Overall, 2,583 referrals were reported over the course of the project, and of these 65.7.0% (n=1,699) were completed referrals from gCHVs to SWs to MHCs. The baseline was zero at the commencement of the project as there was no inclusion of gCHVs in the referral process. Even though the target of 80 percent set was not met, the training of gCHVs greatly increased the number of people successfully referred into care and helped tackle stigma in the communities. The intervention further confirmed the importance of referrals in increasing use of appropriate mental health services, whether it is from communities to primary health facilities, or from primary level of care to secondary level of care. Box 1: Gardnersville Community Clinic Case Study I have been with this sickness for a long time�, says a 28 -year-old Gardnersville resident, who has epilepsy. She said: “I first went to the country doctor, and even to the pastor. I spent lots of money – but nothing helped me�. Last year, I was referred to Gardnersville Community Clinic by a gCHV working in my community. Once there I was screened, diagnosed, and treated, and have been receiving treatment ever since: “when I first came for treatment, I didn’t think it would be better. But it has helped me a lot. I do now feel like I can live my life again� – Lady P. h) Increased supervision and support of MHCs and CAMHCs Working with CHTs to conduct supervision and support helped in institutionalizing quality mental health care in Liberia. A total of 831 supervision visits were made to the 45 health facilities throughout the project. Over the project period, facilities received an average of 18.5 visits with each facility received at least the recommended 2 visits per quarter. But there was significant variation in the number of times any facility was visited. For example, two facilities were visited over thirty times, while several others received less than fifteen supervision visits. Cumulatively, 36.9% of the health facilities received supportive supervision in year 1, 40.4 in year 2, 71% in year 3 and 57% in year 4. Even though the target of 80 percent set was not met due to infrastructure and human resource challenges, the project created a system of supervision and support that did not exist prior to the project, as monitoring capacities increased in the two counties under the project. Page 13 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) During supervision visits, supervisors examined overall strengths and weakness of the facilities and health care workers, as well as observing patient encounters with providers. This created opportunity to provide feedback on how mhGAP-trained health workers interacted with, diagnosed, and treated patients. In addition to the above, monthly case consultation meetings were held with service providers in each of the two counties and provided opportunities for providers to come together and discuss challenging cases that they have encountered, enabling them to learn from each other and improve their skills on handling such cases. The meetings were also used to review important topics, including: quality of care, referral pathways, depression in adolescents, patient outcome forms, and post-traumatic stress disorder. The Project Advisory Committee (PAC) set up also played important role in identifying and addressing issues of data discrepancies between the central ministry and the counties. This effort led to the establishment of a Mental Health Evaluation focal point person at the MOH to monitor mental health data submitted to the Ministry. i) Levels of trust at the community level A total of 131 communities (21 in Margibi and 110 in Montserrado Counties) were reached with at least one intervention (including community healing dialogues, peer support groups, group counseling, and individual counseling). The achievement surpassed target set by 42.8%. Community healing dialogues provided an effective way for EVD survivors with a forum to share their daily struggles during and after the Ebola outbreak and to help one other to rebuild their lives. The impact of the dialogues was conducted on community stigma against Ebola-affected individual/households using a matched comparison study; communities with similar characteristics to those that received the community healing dialogues as a comparison against the intervention communities. The study found that communities that participated in the community healing dialogues had lower levels of stigma than the comparable communities who did not receive this intervention. With higher numbers representing lower levels of stigma, the mean score in the intervention communities was 10.57 or 66.09% (median: 11, standard deviation: 2.64) while the average score for the comparison communities was 9.95 or 62.21% (median: 10, SD: 2.06). There was a statistically significant mean difference of 0.62 or 3.88% (p=.0164). This indicates that the communities that participated in the Community Healing Dialogues had slightly higher levels of community trust than the communities who did not receive the intervention. The training of 200 religious leaders and traditional healers on identifying individuals with acute mental distress, appropriate referral mechanisms, and basic information on mental health, mental illness, and stigma helped to strengthen the system of care and support for persons with mental illness in these catchment communities, including implementation of anti-stigma activities in their communities. Box 2: Cotton Tree Health Center Case Study Mr J, a 35-year old male works as a money-changer. He had suffered from epilepsy for several years, but one day he had a seizure at work. His community took him to sick-bush and to church for treatment, but his seizures continued, and he was unable to work. Due to the stigma of epilepsy, his girlfriend left him. Recently, he heard about the services offered at Cotton Tree Health Center and decided to seek treatment. After three months of treatment for his epilepsy, the frequency of seizures significantly reduced, and he was able to return to work. Page 14 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) j) Individuals receiving direct support (peer groups and individual/group counseling) As evidenced through end-user success stories, the project demonstrated that with the right support, those with mental health issues can live empowered, dignified, and fulfilled lives. Through the project, a total of 1,450 individuals received direct support from peer groups and individual or group counseling. This achievement surpassed end target of 400 by 263%. Of these, almost three-quarters (71.7 per cent) were female. Persons participating in individual counseling were referred in several ways, including via gCHVs and community organizations, as well as through community-based activities such as peer group support sessions or community dialogues. Individual beneficiaries interviewed as part of the evaluation indicated the activity also raised awareness about mental health to families, and to a lesser extent to communities, further helping end-users thriving personally and socially. In addition to the therapeutic benefits of peer counseling, participants confirmed feelings of belonging and reduced stigma. It was also noted by health staff and gCHVs that peer support participants were sometimes able to refer non-participants for treatment. k) Students receiving weekly support from CAMHCs Poor mental health can have negative effects on the wider health and development of children and adolescents. Hence, healthy development during childhood and adolescence contributes to mental health and supports a healthy and thriving population. This indicator tracked the number of students receiving care in the school- based clinics (SBCs), including 4 SBCs (B.W. Payne, Tubman, Lango Lappaye, and KRTTI) funded directly through the project and 3 SBCs ((LMA, R.S. Caulfield, Paynesville Community School) funded through another grant but staffed by CAMHCs trained under the project. The SBCs adopted an integrated care model, whereby mental and physical health care are offered concurrently, without a distinction made between students receiving mental health or physical health care. Students who came for physical care were also assessed for psychosocial issues, and vice versa. This model ensured that students accessed full range of health care needs in a non-stigmatizing environment. Cumulatively, a total of 7,824 students received support through the project. This achievement exceeded the end target of 500 students. Also, this indicator only captured the students receiving support in the 7 SBCs in Montserrado and Margibi project counties and did not capture students supported by CAMHCs trained under this project but working in the other 13 counties in Liberia. A MHC noted: “This intervention has helped to address the perception that children and adolescents with social, emotional and behavioral problems are not “born bad� or “just bad� is new to many people. Even as a MHC, I did not have the competencies that CAMHCs have to relate to children and young people, detect mental health problems early, and treat them with counselling or psychotropic medications until I was trained in child and adolescent mental health�.9 9 FGD, CAMHC, 10 November 2018. Page 15 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Fig 2: Cumulative Breakdown of Students Receiving Care per School 2500 2046 2000 1700 1454 1500 1274 1232 895 868 951 1000 597 653 716 592 468 549 298 406 298 500 167 242 98 144 0 RS Caulfield Tubman High BW Payne KRTTI LMA Lango Lappaye PCS Mental Health Cases Non-mental Health cases The project collaborated with Liberia’s Ministry of Education (MOE), County Education Officers, and the Ministry of Gender, Children, and Social Protection to identify schools in need of CAMHCs. The schools served as the CAMHC training clinical placement sites and established as SBCs through the project. Many trained CAMHCs are based in hospitals and clinics, and some provided regular support to local schools. By the end of the project, 41 CAMHCs were supporting 33 schools. Additional school support was provided to train teachers, school administrators and social welfare workers trained to assess, treat, and manage childhood social, emotional, and behavioral problems, and make referrals; promote positive development; and support learning. For a long-term impact, the project worked with Liberia Board of Nursing and Midwifery and the Liberia National Physician Assistants’ Association, to integrate mental health training into Liberia's national curriculum for nursing schools, and the training of a specialized mental health workforce under the leadership of MOH. A student noted “there are many problems that students can have that require attention to help them behave and do better in Box 3: KRTTI Demonstration School SBC Case Study school. If I feel down or A 16year-old student at KRTTI Demonstration School, dealing with anger issues narrated his upset, I can come to experience as follows: Last year I fought twice in class with other students. I just get very this clinic�.10 Students vexed, and then it happens that I can’t control my anger and what I do. And that’s how the who seek treatment fighting happens. I experienced similar issues outside of school as well. I even fought on old woman here in my community. I believe she insulted me. I was angry, and we hit each other. for physical ailments So, people came to see me as a boy that causes problems. Anger issues at school led to my offer the CAMHC an suspension, and the threat of expulsion. But instead of me being expelled, I was sent to the opportunity to screen SBC to meet with the CAMHC. She worked with me through individual counseling and for social, emotional participated in group counseling. Through the counselling sessions, I learned several anger and behavioral management techniques which I applied inside and outside of school. The first thing I do is problems and conflicts breathing exercises which make me calmer. Also, I learned I can change location, or weigh with families and the pros and cons of a situation. These techniques enabled me to avoid confrontations, be friends that impede more in control of myself, and better my situation. students’ well-being. The CAMHCs at SBCs indicated their belief that this model of service delivery is especially useful for developing a rapport with learners, and long-time contact that consolidates trust in a way that may not be possible at community-based health facilities. A CAMHC noted: “it’s not just about treatment, it’s about getting involved in peoples’ lives�.11 10 KII, KRTTI Demonstration School, 6 November 2018. 11 FGD, CAMHCs, 10 November 2018. Page 16 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Teachers interviewed as part of the evaluation indicated that participating in training gave them increased understanding of importance of mental health and related issues, including: childhood development, behavioral management strategies for the classroom, life-skills education, as well as guidance on identification and referral of students for mental health care to SBCs and other health facilities. In 4 of the SBCs visited during the evaluation, there was a perception that corporal and labor-based punishments have decreased since training. One of the trained teachers said: “We as teachers now understand that learners’ problems are deeper than the classroom, and they need to be helped and not to be punished or made to work�.12 This sentiment was echoed by other teachers that were interviewed. However, there is still an overall persistence of punitive actions that connects to cultural stereotypes of perceived experiences of children in Liberia. l) Percentage of trainers whose trainees demonstrate improvement in their competence/knowledge of women’s health A total of 300 women were reached through this component of the project, 60 women trained as trainers successfully trained four women each, 240 in total, as well as providing them with a health promotion toolkit. Working directly with women proved an empowering model of intervention, with internal project evaluation data indicating increase in feelings of self-efficacy and confidence among the women that were trained. Results of the evaluation revealed that health promotion training and education had significant impact on wellness including increase in self-efficiency, reduction in depression and PTSD symptoms, and increase in self- perception of improved health status as shown in the tables below: Table 2: Depression Symptoms Pre- and Post-test 60 Women trained as 240 Women trained by trainers trainers Pre-test Post-test Pre-test Post-test 13 Experiencing open mole 6.7% 1.7% 11.7% 2.1% Feeling down and depressed 78.3% 38.3% 70% 32.9% Have little interest in doing 78.3% 45% 66.7% 45.4% things Table 3: PTSD Symptoms Pre- and Post-test 60 women trained as 240 women trained by trainers trainers Pre-test Post-test Pre-test Post-test Pain interferes with normal work 75% 33.3% 67.9% 45% Health problems interferes with normal work 68.3% 31.7% 59.6% 41.2% Worry interferes with normal work 63.3% 36.7% 64.6% 43.8% Physical health problems interfere with 55% 38.3% 60% 36.7% normal work Further, 10 of the 300 hundred women were trained in micro-enterprise activity ‘Pad4Girls’ production making reusable and affordable sanitary pads for teens and young women. The training focused on tailoring skills for 12 KII, RS Caufield SBC, 7 November 2018. 13 Open mole is Liberian traditional way of describing depression and anxiety Page 17 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) pad production, menstrual hygiene management, and marketing strategy for sustainability. The inclusion of micro-enterprise training was included as a means of testing options for scaling the project and offer opportunity for project sustainability. m) Direct project beneficiaries The total number of beneficiaries reached by the project was 22,334 amongst which was 54.48% female. This achievement surpassed target by 19%. The achievement recorded includes 7,091 (49% female) individuals treated at facilities for mental illness, 463 (61.8% female) participants in the Community Healing Dialogues, 2,800 (51% female) individuals participating in World Mental Health Day 2016 activities in the communities, 369 (42% female) first responders trained, 1,446 (72% female) people receiving group/individual counseling or participating in peer support groups, 7,824 (58% female) students receiving care at the SBCs, 819 (52.63% female) personnel trained, 865 (54% female) individuals reached through the facility-based anti-stigma training, 300 (100% female) women reached through the training in the women’s health toolkit component, and 357 (51% female) people reached through World Mental Health Day 2017 work place activities. n) Health personnel receiving training A total of 996 health care personnel were trained under the project. They included 80 gCHVs, 71 MHCs, 158 SWs, 12 PSS, 55 OICs, 45 HCWs, 102 CAMHCs, 93 service users, 180 teachers, and 200 religious and traditional leaders. This exceeds the end target of 750 individuals trained by 33%. Many of these individuals received more than one training through the project (see table 4 below). The aim was to develop a core set of skills in identification, treatment and referral for the same health care workers as opposed to providing one-off trainings for a larger group of workers. The strategy was to embed holistic set of skills in the health care system, rather than just providing individual trainings to more people. In addition, the project developed skills in community support, peer support for para-professionals, users, and community leaders. Table 4: Trainings Completed by Type of Workers Type of Worker Trainings Received Total number of individuals trained General Community Health Volunteers Anti-stigma, Community Informant Detection Tool (CIDT), 80 (gCHVs) Psychological First Aid (PFA), Resilience training Mental Health Clinicians (MHCs) Counselling and listening skills, Facilitating Community 71 Healing Dialogues (CHDs), Interpersonal group therapy, Facilitating support groups, Substance Use Disorder training, Resilience training Social Workers (SWs) Family Psychoeducation (FPEM) and counselling and 158 listening skills, Facilitating support groups, Resilience training, EMRO Manual of School Mental Health Training of Trainers Psychosocial Workers (PSS) Family Psychoeducation (FPEM) and counselling and 12 listening skills Officer in Charge (OICs) mhGAP 55 Mid-level Health Care Workers (HCWs) mhGAP, Resilience training 45 Child and Adolescent Mental Health Post-Basic C&A MH course, EMRO Manual of School 102 Clinicians (CAMHCs) Mental Health Training of Trainers, Children’s Accelerated Trauma Therapy (CATT) Service Users (Peer workers) Facilitating peer support groups 93 Page 18 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Teachers Family Psychoeducation (FPEM), EMRO Manual of School 180 Mental Health Training of Trainers Religious and Traditional Leaders Acute stress identification, anti-stigma, collaborative care, 200 and referral pathways Total 996 The final independent evaluation of the project done by TCC revealed that the project made important contributions to building skills in mental health through trainings including clinical competency in managing mental cases, address burnout, proper record-keeping and data management. The training also provided tangible number of MHCs with each of the 15 counties in Liberia now have at least three MHCs. This has contributed to national expansion of mental health interventions which is a vital outcome of this project. The trainings capacitated clinicians to go beyond providing physical care but also, exploring underlying causes and providing appropriate mental health care to patients. As indicated by one trained health care provider, “before I would treat patients for other disease conditions – like typhoid and malaria. But now I know enough to probe about their life and their problems. So, the training has enabled me identify cases of depression and other mental health cases�.14 As said by another, “I would see patients again and again; and still I would send them away with medication for physical illnesses. Now I have greater awareness about mental health issues, and able to provide holistic treatment of mental and physical ailments�.15 Box 4: Dolo Town Clinic Case Study An 18-year-old client interviewed shared his experience as follows: My parents took me to numerous traditional healers and health facilities in Margibi for treatment for my seizure ailment. I was given many pills but never worked. Even the country doctor tried using herbs, yet no result. I stopped school, because of fear of falling due to seizures. No one or myself knew what was happening to me. My parents took me to Freetown in Sierra Leone to seek treatment with no effect. After returning to Liberia, a neighbor suggested that I seek treatment at the Dolo Town Clinic, where a project- trained MHC successfully diagnosed and treated another boy of my age with epilepsy. It was at the clinic that my ailment was diagnosed as epilepsy. Since then I have been receiving treatment and have not experience an epileptic attack. I can now concentrate at school, play football and partake in any activity with no fear. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME 25. The following are factors noted to have affected project implementation negatively and extent to which project outcomes ought to have been achieved: 14 KII, New Destiny Clinic, 6 November 2018. 15 KII, Redemption Hospital, 8 November 2018. Page 19 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) a) Shortage of psychotropic drugs: There was programmatic assumption that National Drug Service (NDS) can supply health facilities, including SBCs, with necessary medications, but this was not the case. The project increased demand for mental health services and increased pressure on health facilities to provide psychotropic medications, and it was beyond the scope of the project to support Liberia’s National Drug Service (NDS) to address shortage attributed to lack of a functional supply system and unwillingness to import psychotropic medicines as part of essential medicines. Thus, undermining the long-term demand for mental health services, and possibly discourage future health-seeking endeavors if they cannot access medications, or their preferred Box 5: Gardnersville Community Clinic Case Study medications. As a result, there was a noted level of relapse among existing patients due to lack of or Lady P confirmed that the Carbamazepine she received inadequacy of medications. The case study originally was effective, but inconsistent supply of the drug created difficulties. She said: “sometimes the drug is not presented in Box 5 below confirmed that inadequate available and when I get other medicines, the seizures can or inconsistent drug supply can decrease demand for come back. This gets me discouraged�. mental health services, as patients have their treatment interrupted. Generally, supply chain is a persistent issue in the country and partners are working with the government to resolve the issue. But addressing issues related to national supply of psychotropic medications in a sustainable manner must be priority for the GoL and its partners. b) Limited resources for supervision and incentives payment: There were challenges in conducting supervision in hard to reach health facilities, as well as the timely collection of data. Such visits require travelling long distances with some areas only accessible by motorbike thus making the activities costlier and more time- consuming. Unlike many health programs that have resources, the mental health project did not have specific vehicles for facilitating supervision and accessing data collection. The issue of incentivizing data collection to make it a priority for health workers posed potential problems. It was noted that if data collection is not incentivized, quantity, quality, and timeliness of monitoring activities will decrease. Despite this challenge, it was also noted that government health data on mental health were more in Montserrado and Margibi Counties than the other counties. c) Coordination with government hierarchies: Even though the project constituted project advisory committee consisting of CHT that met quarterly and addressing issues and challenges, the mental health unit at the central MoH felt that coordination became ineffective following approval and planning for the project at the central level. An indication that coordination of implementation of project activities was limited to the counties and not communicated to, and coordinated with, the central Ministry. This has implications for full integration of the gains of the project. The World Bank worked closely with the mental health unit at the central MoH and raised the issue of sustainability at the highest-level policy meetings advocating for these measures. d) Limited referrals from religious and traditional leaders: despite the training of religious and traditional leaders, only few referrals were noted as traditional and religious leaders perceived themselves to be qualified to undertake mental health counseling, instead of simply identifying and referring cases. Patients were often brought to health facilities after traditional or spiritual ‘care’ options were exhausted. As stated in one discussion: “Not all cases can be referred. There are some types of cases that only need spiritual counseling. Severe cases with violence, and epilepsy, and deep depression should go to the clinic�. Most religious leaders believed that mental health care requires a combination of medical treatment with spirituality. They believed patients need spiritual counseling, “without God, even the treatment wouldn’t work. God is key� commented by a religious Page 20 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) leader. In addition, the unlikelihood of traditional healers referring patients was noted. They associated this to depriving them their means of livelihoods. Based on the above, it may be important to engage the appropriate institutional structures of the Liberian Council of Churches, National Muslim Council of Liberia, Traditional Medicine Federation of Liberia, etc. Their participation may provide greater authority to institutionalizing mental health messaging in the long-term into their activities and serve as agents of referral. In addition, the limited referrals may be addressed by adopting user-led referral. Empowering patients as ambassadors for mental health referral, offers an opportunity for greater project impact and sustainability. e) Limited number of gCHVs for catchment areas: The community-based activities of gCHVs are important to anti-stigma and referral. But the catchment areas were too large for allocated 1-2 gCHVs to cover. The project revealed that referrals require considerable work from community-based staff as imparting information on mental health and services was not enough to complete referrals. Successful referrals often require a number of follow-up visits before patients or caregivers can be convinced to seek care at a facility. Often patients would need to be accompanied to facilities, and at times requested funds for transportation. Given the level of engagement required to complete referrals and the fact that gCHVs cannot be incentivized to continue providing support, it is likely that this intervention would decrease after project completion. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. RECIPIENT’S OVERALL PERFORMANCE 26. The Carter Center was very committed to achieving the objectives of the project and put measures in place as needed. They worked with stakeholders in country to implement the project. This gave in-country stakeholders sense of ownership including the beneficiaries. TCC provided the support needed and developed capacity of stakeholders in country to manage the project. International and local mental health experts were involved in project implementation, monitoring and evaluation. The experts played key roles in developing training curricula for all cadres of mental health care providers; conducted trainings and provided technical assistance as needed. To ensure technical and institutional sustainability, TCC worked closely with the MOHSW, CHTs in the two counties of Monsterado and Margibi, the parliaments, and leverage on activities of other development partners, thus maximizing use of resources. 27. TCC was very responsive to implementation issues such post procurement review recommendations and responded adequately to feedback provided on technical and financial quarterly reports. These enabled them to ensure compliance with the project covenants. B. QUALITY OF MONITORING AND EVALUATION (M&E) 28. The overall rating for M & E for the project is Substantial, owing to efforts and measures put in place despite the challenging environment in ensuring delivery of quality project. Implementation of activities focused on achieving the desired results and project development objectives. Due to monitoring efforts in place, the project was able to detect challenges on time and took adequate measures including the implementing partner taking on responsibility for incentives payment to obtain timely data collection of the mental health services provided. Page 21 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Even though the project achieved the PDO indicators, the setting of target for one of the 3 PDO indicators was inappropriate. A lower target below the baseline would have been appropriate to measure reporting of reduction in clinician stigma as detailed in the report. Lesson from this should be taken into cognizance in designing indicators and setting targets for future mental health projects. 29. The project conducted regular technical review meetings especially following review of quarterly progress reports. The meetings enabled the project to collectively address issues and challenges identified in the reporting period and clarify any observations noted as much as possible. This was in addition to field visits conducted to trainings to ensure technical quality, observed staff while conducting some non-confidential interventions, and interacted with clients on services received. 30. A Project Advisory Committee (PAC) was constituted to play oversight role for the project, promote ownership and working with the implementing partner in ensuring delivery of outputs and targets, including resolving challenges and issues that could impact negatively on the project target outputs. The PAC consists of representation from the MoH, CHT, the Carter Center and community heads. The PAC met regularly to assess project progress and attended adhoc meetings where necessary. Recommendations from those meetings contributed to resolving challenges faced by the project. C. ENVIRONMENTAL, SOCIAL AND FIDUCIARY COMPLIANCE 31. Financial Management (FM): The FM was rated Moderately Satisfactory over the life of the project given that: (1) the funds for the project are used for the intended purposes with due regards for economy, efficiency and effectiveness, (2) timely and reliable financial reports for the project are produced, and (3) measures were in place to protect the assets of the project. There are no outstanding Interim Financial Reports (IFRs) or audit reports under this operation. The IFRs have been submitted in good condition and in the appropriate format although in some instances the IFRs will be submitted a day or two late. The audit reports for all the fiscal years were submitted on time as specified by the Legal Agreements and the audits opinion were all expressed with unqualified opinions on the project financial statements. 32. The overall FM arrangements were in place including staffing, controls, accounting, reporting and auditing. The FM arrangements were unique in nature as the project had no designated account but was using the same account that Cater Center uses for a donor funded project. The rating for the purpose of the ICR is Satisfactory 33. Procurement: Overall procurement performance under the project was Moderately Satisfactory. The Carter Center (TCC) a Non-Governmental Organization (NGO) which was selected as the Implementing Agency (IA) lacked previous experience in World Bank Procurement Procedures. This situation was worsened by the fact that it was difficult to implement the mitigation measure for the Bank and the Ministry of Health (MOH) Procurement Specialists to provide hand-holding support to TCC procurement staff. This was because instead of TCC Monrovia taking a lead in facilitating procurements, their headquarters in Atlanta, Georgia, USA carried all procurements under the Project. To address this the Bank procurement team held video conference meetings to discuss procurement implementation issues and closely monitored the implementation of mitigation measures recommended from Procurement Post Reviews (PPRs). This intervention enhanced implementation. Page 22 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) BANK PERFORMANCE 34. The ICR rates Bank performance over the course of project implementation as Satisfactory. The rating is largely due to the Bank’s proactivity with supporting TCC, the project’s PDO and components, as well as RF, were aligned to focus on the priority needs of the country. Even though an International NGO – the Carter Center implemented the project, the World Bank team ensured closing working relationships with the mental health unit of the central Ministry of Health and continuously advocate for mental health services at the various high- level policy meetings towards effective integration and sustainability. 35. Overall, the project supervision by the Bank team was very thorough and undertaken adequately. The engagement of the implementing partner and the MoH was strong through the project implementation period. Frequent meetings of the government (MoH and CHT) and implementing partner helped to discuss progress, results, challenges and bottlenecks with adequate measures taken to address promptly technical and operational issues in the planned activities to achieve targets and project development objectives. The Bank’s TTL accommodated flexibility by moving funds from consulting and goods to training and operational costs to fulfill the financial needs of the project towards achieving the expected results. Also, the Bank’s flexibility and creativity demonstrated during implementation were essential for the achievements of the project results in an FCV context. 36. The shortcoming noted during implementation was in the initial reporting which lacked linkage of activities implemented with realistic results on the ground. This was discussed and led to inclusion of some qualitative information of the changes happening due to the trainings and community interventions implemented. Even though, the RF was revised during level 2 project restructuring, there were still challenges with the targets set for three of the indicators. These were noted in the report. Despite this, the PDO was achieved. V. LESSONS LEARNED AND RECOMMENDATIONS 37. The following lessons learned and recommendations from the project are intended to inform future World Bank financed operations, especially for those being implemented in an FCV context. 38. Selection of appropriate indicators: Experience managing this project revealed that it was difficult to interpret few of the indicators especially when it comes to measuring progress with the indicators. For example, some indicators were measured one off and could not track progress following implementation of interventions to improve on the first measurement despite the positive impact of the interventions. Also, few indicators in which lower targets were to be set (as indication of improvement or positive changes) had higher targets set than the baseline. Defining indicators to measure and track progress require clarity in the how it should be measured to track the changes happening because of project interventions. This needs to be considered in future design of projects of this nature. 39. Incorrect assumption of getting supplies from NDS: The project was designed with the intention that psychotropic drugs would be made available for the treatment and management of mental health cases without considering the challenges with drugs and supply chain faced by the country. The inconsistent supply of Page 23 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) medications affected the extent of the impact of the project as noted under factors that affected implementation. Future project design may need to firm up agreement with the GoL to ensure consistent supply of psychotropic medications or the project allocate resources to cover provision of drugs. 40. Incentivizing activities implemented: As the project progressed, payment of incentives became a major issue. While it is understandable to incentivize the gCHVs for their work in communities, expectation of incentives for data reporting of mental health services provided was very surprising and unsustainable. This is a key lesson that needs to be considered in future projects. Before health care providers are selected for training, readiness to carry out all the activities expected of them must be a pre-condition for selection for training, while readiness of government to take on volunteers’ incentives to enable them to continue what they have been trained to do needs to be negotiated and agreed. In addition, subsequent designs of the project could encourage ownership by communities to periodically raise funds as incentives for the gCHVs to sustain their work in the communities. 41. Benefits to young people including out of school youth: The impact of SBCs in detecting mental health issues amongst students in participating schools have been elaborated. The out of school youth were excluded from this intervention. Future design may need to look at how such groups could be captured to benefit from psychosocial interventions considering Liberia being a post conflict country with large number of out of school youth. This could help improve productivity and enhance youth positive behavior. This will require working effectively with the Ministry of Education (MoE), Ministry of Youth and Sport (MoYS) and the training institutions to ensure effective integration of mental health into sectoral plans for continuity and sustainability. 42. Lack of linkage to World Bank Health Portfolio in Liberia: The project was designed as a stand-alone and implemented by an entity outside of GoL to respond to psychosocial crisis caused by Ebola viral disease (EVD), and not linked to the existing health portfolio supported by the World Bank. Despite the relevance and achievements recorded by the project, it has not attracted funding from other donors including the World Bank as psychosocial health was not presented as priority by the GoL. Future project design needs to consider dialoguing and engaging with government and ensure that the project responds to their priority(ies). This could promote government commitment to continuous funding and present it as priority to their development partners and donors for financial and technical support.. Sustainability 43. This section outlines important gains in project sustainability, as well as a number of important challenges to sustaining project effectiveness and impact, including: lack of government funding for mental health, competing health priorities, lack of incentives, staff turnover and attrition, and effort on-going to continuously engage the government on GoL budgetary allocation to mental health interventions. 44. Sustainability Gains: Supporting psychosocial health and resilience project in Liberia created an important blueprint for what could be replicated for building up mental health capacities in the future. The project was the first comprehensive model of mental health service delivery in the country. Evidence of sustainability is present in the fact that some project activities are still on-going in health facilities and project schools as a result of staff capacity already built and support from TCC, despite decreased direct financial and technical assistance. Most notably, MHCs and CAMHCs who were trained to provide mental health interventions for individuals and groups have continued to provide these services in the counties. Training received through the project helped to support the continuing education and further development of clinicians’ skills, reinforcing capacities for mental health care, and helping to expand and strengthen mental health services around the country. Government Page 24 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) participation in project design was important in this regard, as the human resources developed through the project will continue to be maintained under the existing health system. This was confirmed by the MoH that: “the project is using existing workers, which are on payroll. So, the government will continue to support these workers to enable them use skills they have gained from the project�.16 45. It is significant to note that Liberia’s new Mental Health Policy 2016 – 2021 explicitly mentions the Phebe School of Nursing as a designated training center for MHCs and commits to further training of MHCs specializing in child and adolescent mental health, presumably at Zwedru Midwifery School.17 However, it is unclear how these training activities will be funded and sustained. In the short-to-medium-term, two cohorts have already been completed at Phebe School of Nursing with funding from John Snow, Inc., and training of a third cohort is underway. The Carter Center has committed to supporting the funding and technical assistance for Zwedru Midwifery School for the first cohort, to get the training integration started, while the mental health unit of the MoH is working towards getting a GoL budget line for mental health. This commitment if achieved will contribute to, the long-term sustainability of mental health capacities and activities implemented. Box 6: Peer support groups efforts in sustaining the gains of the project Some peer support groups have continued to meet, even though they no longer receive direct funding or support from the project. In some cases, such groups had been raising their own funds to support meetings and emergency drug purchases. For example, the Peer Support Group connected to Cotton Tree Health Center collects LRD 25 (16 cents) from each member every meeting to support treatment in case medications falls short at the health facilities by purchasing medications from private sources. The initiative taken by these peer groups indicates proactive efforts that extend beyond planned project activities, and which could be harnessed to promote sustainability. One strategy for promoting sustainability of peer support may be to formalize the groups by registering them with the government and orientating them towards raising awareness about mental health issues at community level while government works on taking over incentivizing and oversight of gCHVs. 46. Contributions of the project to drafting of the Mental Health Policy helped to further institutionalize GoL commitments to mental health. However, there is no clear complementarity between efforts to improve legislation and programming activities. The Carter Center was also instrumental in the passage of the Mental Health Law that forms part of the Public Health Law. It is Liberia’s first law to protect the human rights of people with mental illness and promises access to mental health care in all fifteen counties. However, much remains to be done including regulation, and the administrative legal scaffolding for the full impact of the law to be experienced. Despite the above efforts made to sustain the project, the following challenges pose threats to project sustainability: 47. Lack of funding for mental health and competing priorities: Implementing mental health in Liberia is almost exclusively contingent upon donor funding. There are signs that the donor funding that Liberian GDP largely dependent upon may decrease in the coming years.18 Should international funding decrease, then it is unlikely that the country will be able to continue to address its mental health objectives. Also, mental health in Liberia is 16 KII, MOH, 2 November 2018. 17 GoL, 2016, Mental Health Policy and Strategic Plan for Liberia 2016-2021. 18 African Development Bank, 2017, “Liberia Economic Outlook�, https://www.afdb.org/en/countries/west-africa/liberia/liberia-economic-outlook/ (22 November 2018). Page 25 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) a nascent health issue and is caught among many other competing government and donor priorities. Thus, for funding that does go to health services, mental health has to compete with other health priorities and there is still a great need to build and sustain awareness among government stakeholders about the need for more expansive quality mental health services. In addition to having mental health unit within the MoH, it is important to consider incorporating mental health indicators into the performance-based contracting health programming or ear-mark a proportion of all health funding to support mental health service delivery in the country. 48. Staff turnover and attrition: Turnover of ministry staff was identified as a future threat to sustainability. The impact of turnover was evidenced following the 2017 Liberian presidential election; new ministerial staff were appointed at the MOE and MOH, both at the central ministries and in the counties. It took additional time to familiarize these new staff with the project. However, as the project is completed, this type of advocacy may no longer be possible. If further changes take place, and high-level champions of mental health in government are replaced, the GoL’s growing commitment to mental health may also wane in the midst of the many competing health and development priorities facing Liberia. Already, the mental health unit within the MoH is working with the leadership of the Ministry to ensure that staff are retained as much as possible. 49. Another challenge noted is that a small number of those trained have moved. If staff move to other health facilities, then the capacities they acquired through training will move with them within the health system – though they will no longer be embedded in and supported by a systemic community-to-facility approach to mental health provision. The worst-case scenario is having a large number of trained mental health staff leave the Liberian health system altogether. Indeed, without support from the government, and extra incentives provided to support facility-based mental health activities, staff attrition may increase. It is important for government to take cognizance of this and ensure strategies are in place to reduce attrition of mental health clinicians trained as much as possible. 50. Lack of incentives: Most gCHVs are still unregistered volunteers whose incentivized activities are contingent upon project funding. Without continued funding, it is unlikely that community level activities will continue. Both gCHVs and other project stakeholders indicated that many health volunteers have been trained and incentivized by previous health projects to support various community health issues, only to have their work cease after project activities and funding stopped. These experiences indicate the tenuous nature of the sustainability of any programming model that relies on referrals from gCHVs. To overcome this challenge, the mental health unit of the MoH is actively engaged in the budgetary process to ensure budget allocation for mental health activities in FY 19/20 budget cycle. Page 26 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: The project development objective was to respond to the intermediate psychosocial and mental health impact of the EVD crisis and to build long-term psychosocial health and resilience at the individual Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Level of competence, skills and Percentage 15.00 55.00 66.00 67.70 confidence among providers 28-Jan-2016 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): As noted in the report, this indicator demonstrates a challenge experienced with the format of the results framework – the values for each of the year do not represent one cohort of individuals improving or worsening over time, but rather an accumulation of scores of different cohorts. Therefore, it is useful to compare the baseline of 15% to the final value of 67.7%, but it is not useful to compare the years against one another as documented in the results framework. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Improvement in Depression, Percentage 0.00 20.00 40.00 66.00 Post-Traumatic Stress Disorder (PTSD), and disability among 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Page 27 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) project beneficiaries Comments (achievements against targets): The scores here represent the number of patients who demonstrated improvement in symptoms of depression, as represented by a drop in their PHQ9/PHQ9A score from baseline to follow up. Like the above indicator, it is not useful to compare years against one another as there was not one set of individuals improving over time. Rather, additional patients were added each quarter as they were treated at the facilities. The achievement surpassed target by 16.5%. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Clinicians stigma against Percentage 43.18 0.00 50.00 17.02 individual with mental illness 01-May-2017 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): Unlike the other scales, a lower score here is desirable, as a lower score indicates lower levels of stigma. The end target here should have been adjusted after the baseline was measured to set a new goal when the levels of stigma were lower than anticipated. Given the baseline of 43.18%, a more reasonable target would have been to have stigma levels at or below 25%. As with the above indicators, the project was not able to track one group for the whole project. There are only scores in years 1 and 2 because the anti-stigma training was carried out early in the project, and not able to collect an additional follow up score with the trained individuals. A.2 Intermediate Results Indicators Component: Component 1: Support for Intermediate Mental Health/Psychosocial Needs Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Page 28 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Levels of stress and burnout Percentage 84.21 10.00 60.00 82.48 among health workers (including first responders) 28-Jan-2016 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): As with the above indicators, the values for each year represent different cohorts of individuals rather than following one group of individuals over multiple years. As a lesson learned from this project, this indicator did not appropriately capture the impact of the self-care and stress relief training for first responders. The training was delivered over two days and the baseline was taken at the beginning of the first day and the follow up score was taken at the end of the second day, which did not provide enough time to analyze how the levels of stress and burnout changed once the first responders implemented the strategies they learned during the training. Due to the transitory nature of many of the first responders, the project was not able to conduct a follow up assessment several months later to assess the true impact of the techniques they learned. A qualitative assessment might more fully demonstrate the impact of this intervention. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Levels of reported high risk Percentage 67.20 40.00 50.00 59.80 health behaviors among responders 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator, as with the above, was used to assess the impact of the self-care and stress relief training for first responders and may not have fully captured the impact of the training. Even though the figure showed surpassed target, improvement and well-being ought to reflect lower target. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Students receiving weekly Number 0.00 100.00 500.00 7824.00 Page 29 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) support from CMCHs 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator tracks the number of students receiving care in the school-based clinics (SBCs), including 4 clinics directly funded through this project and 3 clinics funded through another grant but staffed by CAMHCs trained under this project. This indicator only captured the students receiving support in the 7 SBCs in Montserrado and Margibi counties, so it is not capturing the students supported by CAMHCs trained under this project but working in the other 13 counties. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Women's health trainers Percentage 0.00 100.00 50.00 100.00 whose trainees demonstrate improvement in their 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 knowledge of women's health Comments (achievements against targets): This indicator relates to the implementation of Women’s Health Toolkit. It tracks the percentage of the women trainers (total of 60 women trained as trainers) whose 240 trainees demonstrate improvement in their competence/knowledge of women’s health. All demonstrated knowledge of women’s health. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Health personnel receiving Number 0.00 175.00 750.00 996.00 training (number) 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator tracks the total personnel receiving training under the project. Many of the 996 persons trained had received more than one training from the project, as the project sought to build a strong cadre of health care workers Page 30 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) with deep experience in mental health. This surpassed achievement by 32 percent. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Direct project beneficiaries Number 0.00 18930.00 18903.00 22334.00 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Female beneficiaries Percentage 0.00 0.00 60.00 54.48 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator tracks the total number of project beneficiaries reached. The total reached was 22,334. This number includes individuals treated at facilities for mental illness (7,091), participants in Community Healing Dialogues (463), community participants in World Mental Health Day 2016 and 2017 (3,157), trained first responders (369), people reached through group/individual counseling and peer support (1,446), students receiving care at the SBCs (7,824), personnel trained in mental health (819), individuals reached through facility-based anti-stigma training (865), and women reached through the women’s healt h toolkit (300). Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Levels of perceived social Percentage 71.00 60.00 78.00 85.00 support 30-Nov-2017 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator as it was introduced later in the project to assess the impact of the recovery groups (Alcoholics Anonymous and Narcotics Anonymous) on the participant’s perceived social support. The baseline was taken at the beginning of the groups and the follow up score was taken at the close of the project. The values represent the percentage of individuals who demonstrated high levels of social support using the Multidimensional Scale of Perceived Social Support (MSPSS). The original target was Page 31 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) revised following a baseline score of 71. The achievement surpassed target by 9%. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Individuals receiving direct Number 0.00 0.00 400.00 1450.00 support (peer groups and individual/group counselling) 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator measures the number of individuals in the project communities receiving direct support through peer groups or individual/group counseling. The baseline was zero as the peer groups and the counselling outreach in these communities began with this project. Project surpassed target set by 263%. Component: Component 2: Support to Build Long-Term Psychosocial Health and Resilience at the Individual and Community Level Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Completed referrals from gCHV Percentage 0.00 100.00 80.00 65.70 to SWs to MCHs 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): The baseline for this indicator, which measures the percentage of total referrals that were completed referrals from a gCHV to MHCs, was zero because there was no inclusion of gCHVs in the referral process prior to this project. It is important to note that this percentage only includes the referrals coming from gCHVs and did not capture referrals from traditional and religious leaders or peer referrals. While the target of 80% was not attained due to challenges discussed in the report, the training of gCHVs greatly increased the number of people successfully referred into care. Indicator Name Unit of Baseline Original Target Formally Revised Actual Achieved at Page 32 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Measure Target Completion Communities receiving Percentage 0.00 0.00 75.00 100.00 community-level support 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator measures the percent of the target communities and the total number of communities reached through the project. At the outset of the project, 131 communities in Montserrado and Margibi counties were identified as target counties for community-level interventions, with the aim of reaching 75% (98) of these communities. The project surpassed the target set by 25% as all 131 communities were reached. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Students receiving weekly Number 0.00 100.00 500.00 7824.00 support from CMCHs 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator tracks the number of students receiving care in the school-based clinics (SBCs), including 4 clinics directly funded through this project and 3 clinics funded through another grant but staffed by CAMHCs trained under this project. This indicator only captured the students receiving support in the 7 SBCs in Montserrado and Margibi counties, so it is not capturing the students supported by CAMHCs trained under this project but working in the other 13 counties. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Women's health trainers Percentage 0.00 100.00 50.00 100.00 whose trainees demonstrate improvement in their 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 knowledge of women's health Page 33 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Comments (achievements against targets): This indicator relates to the implementation of Women’s Health Toolkit. It tracks the percentage of the women trainers (total of 60 women trained as trainers) whose 240 trainees demonstrate improvement in their competence/knowledge of women’s health. All demonstrated knowledge of women’s health. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Individuals receiving direct Number 0.00 0.00 400.00 1450.00 support (peer groups and individual/group counselling) 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator measures the number of individuals in the project communities receiving direct support through peer groups or individual/group counseling. The baseline was zero as the peer groups and the counselling outreach in these communities began with this project. Project surpassed target set by 263%. Unlinked Indicators Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Supervision and support of Percentage 0.00 0.00 80.00 57.00 MHCs and CHMCs 25-Feb-2015 25-Feb-2015 31-Dec-2018 05-Apr-2019 Comments (achievements against targets): This indicator, which measures the percentage of the 45 project facilities receiving at least two supervision visits per quarter, has a baseline of zero because at the time that the project started, there was no role of senior MHCs and CAMHCs who were conducting supervision at facilities providing mental health services. Target set for the indicator was not attained due to resource challenges detailed in the report). However, the project succeeded in establishing system of supportive supervision that did not Page 34 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) previously exist. Page 35 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) A. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1: To respond to intermediate psychosocial/mental health impacts of the Ebola crisis and to build long-term psychosocial health and resilience at the individual and community level in project target areas. 1. Level of competence, skills and confidence among providers Outcome Indicators 2. Improvement in depression, PTSD, and disability among project beneficiaries 3. Clinician stigma against individuals with mental illness 1. Levels of stress and burnout among health workers (first responders) 2. Levels of reported high risk health behaviors among responders 3. Levels of Perceived Social Support 4. Completed referrals from gCHVs to SWs to MHCs 5. Increased supervision and support of MHCs and CAHMCs Intermediate Results Indicators 6. Communities receiving community-level interventions. 7. Individuals receiving direct support (peer groups and individual/group counseling) 8. Students receiving support from CAMHCs 9. Percentage of women’s health trainers whose trainees demonstrate improvement in their knowledge/competence of women’s health topics. 10. Direct Project beneficiaries 11. Health Personnel receiving training Component 1: Support for intermediate psychosocial/mental health impact of the Ebola crisis Outputs Key Outputs by Component 1. 369 first responders trained on self-care and stress relief (linked to the achievement of the 2. Ten recovery groups (14 participants in each group) established Objective/Outcome 1) 3. 1,450 individuals benefitted from individual and group counselling 4. 33,402 patient encounters recorded, 29,114 mental health cases diagnosed and managed. Page 36 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) Component 2: Support to build long-term psychosocial health and resilience at the individual and community level Outputs 1. 996 selected cadres of providers trained on various topical areas as documented in table 4. 2. 41 CAMHCs deployed and providing support to 33 schools 3. 4 school-based clinics established 4. 7,824 students received care through the project including 52% females. 5. 300 women trained in health promotion tool kit Page 37 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) . ANNEX 2. PROJECT COST BY COMPONENT A. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1 0.51 0.51 100 Component 2 1.99 1.99 100 Component 3 0.25 0.25 100 Page 38 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS None Page 39 of 39 The World Bank Supporting Psychosocial Health and Resilience in Liberia (P146591) ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) 1. Quarter 15 (last quarter project report) Q15 TCC Progress Report to WB - Final.docx 2. Project Evaluation Report. Evaluation for Carter Center - World Bank - 18_12_21 FINAL.docx Page 40 of 39