49432 Core Course: China Regional Course on Health Sector Reform and Sustainable Finance Ray C. Rist WBI Evaluation Unit World Bank Fumika Ouchi Advisor Evaluation Analyst (202) 458-5625 (202) 473-5085 Number 20 October 1998 Johansen, Anne (Notation: The World Bank Institute was formerly called the Economic Development Institute (EDI), as reflected in some text) The Human Development Division of EDI (EDIHD) conducted a training-of-trainers (TOT) course on Health Sector Reform and Sustainable Finance in Shanghai, China, from June 1- 17, 1998. This was the first regional course adapted from the Health Flagship course held earlier in Washington, D.C. (1997). Three modules from the Washington course, Modules 2, 3 and 4, were offered. The course focused on diagnostic approaches to assessing health systems (Module 2), revenue sources and collection modalities (Module 3), and targeting public subsidies for health (Module 4). Fifty-nine participants attended Module 2, 60 participated in Module 3, and 61 in Module 4. The participants consisted mainly of faculty members from the China Network for Training and Research in Health Economics and Finance, as well as local health administrators and graduate students from the Department of Health Economics at Shanghai Medical University. The course evaluation was supported by the EDI Evaluation Unit (EDIES). Results from end- of-module (EOM) questionnaires were processed by the Shanghai Medical University, and used in the evaluation. These questionnaires were completed by 45 respondents in Module 2 (76.3%), 43 respondents in Module 3 (71.7%), and 41 respondents in Module 4 (67.2%). There were 20 women among the 45 respondents in Module 2 (44.4%), 23 among the 43 respondents in Module 3 (53.5%), and 20 among the 41 respondents in Module 4 (48.8%). Thirty-seven respondents in Module 2 had previous training in health economics, 39 in Module 3, and 34 in Module 4. The questionnaire used to evaluate each Module consisted of 2 parts. The first section asked respondents to rate the overall design and delivery of the Module. The second section asked the level of respondents' knowledge about topics covered in the Module, before and after completing the Module. A 6-point Likert type scale that ranged from 1 = minimum to 6 = maximum was used to rate respondents for each question. Following is a summary of the evaluation findings. l Results from all three Modules showed high respondent ratings on two key areas of the course performance. These were the relevancy of materials to current occupation and the overall use of respondents' time. Module 2 had the highest mean scores on these two indicators among all modules, 5.38 and 5.53, respectively. The ratings for the other two modules also exceeded 5.0 on a 6-point scale, 5.19 and 5.27 for Module 3 and 5.02 and 5.29 for Module 4. l The course was designed to expose participants to knowledge and skills of both international and Chinese experts for each Module. The trainers of Module 2 received particularly favorable responses from respondents for their clarity in delivering their messages. The mean score for this indicator was 5.33 out of 6.0, the highest rating among all indicators for trainers. Respondents also gave high approval ratings to the Module 3 trainers for their ability to answer questions adequately and the level of their knowledge. Mean scores were both 5.21. knowledge. Mean scores were both 5.21. lChina-specific cases were used in each module to enhance participants' understanding of issues. Respondents were asked whether the Chinese cases were relevant to the teaching objectives, and whether they contributed to respondents' understanding of the theories and methods. The mean scores for Module 4 were the highest among Modules, 4.84 and 4.86, respectively. Module 2 had the lowest ratings among the three Modules, 4.18 and 4.36 out of 6.0. lThe lowest course performance indicator across modules was on the degree to which the topics were treated in sufficient depth. The mean scores on this indicator for Modules 2-4 were 4.33, 4.56, and 4.39, respectively, out of 6.0. lPre/post knowledge assessment showed an overall increase in respondents knowledge among all modules and items. Module 4 had had particularly large gains. The pre-course ratings of Module 4 ranged between 2.82-3.63 and the post-course ranged between 4.63-5.15. The percentage gain was more than 64% on two key topics, the information required to analyze incidence in health and the mechanics of incidence analysis in health. Three other items had a percentage gain of over 50%. lThere was a significant variation in pre/post percentage gains among issues covered in Module 2. Issues on health system structural components had the highest pre- course rating, 4.11 out of 6.0. The percentage gain on this topic was 23.8%, the lowest gain in Module 2. The complexity of causes involving rational economic reform had the lowest pre-course rating of 3.18. The rating of this topic rose by 55% upon completion of the Module, to the highest post-course mean score in Module 2, 4.93. Module 3 had a relatively even distribution of percentage gains across issues, about 30-40% across topics. The pre-course mean scores ranged between 3.24-3.83. The post-course range was 4.74-5.05. lThere were three limitations to the evaluation. First, the data file of the original EOM questionnaire was kept in China and was not available for the evaluation. Therefore, results were not analyzed by detailed demographic / professional categories. Second, the information obtained from pre/post knowledge assessment reflects the degree to which respondents believed they learned the topics, but does not measure how much they actually learned. Third, the 6-point measurement scale lacks a mid-point, and thus the results may not be reliable. The use of 5-point scale with both a mid-point category and a "no answer" option is suggested for the future courses.