Evaluation of FY02-03 WBI Activities with Indonesian Participants: Relevance, Effectiveness and Impact Jaime B. Quizon WBI Evaluation Studies EG05-96 The World Bank Institute The World Bank Washington, D.C. July, 2004 ACKNOWLEDGEMENTS This report was prepared for the World Bank Institute (WBI) under the overall guidance of Marlaine Lockheed, Manager, WBI Evaluation Group. Jaime Quizon of the Institute Evaluation Group (IEG) prepared this report. The author thanks Anders G. Zeijlon who served as peer reviewer of this evaluation study and offered many suggestions for its improvement. The author also thanks Denis Nikitin for his assistance with the statistical analysis, and Humberto Diaz for his help with formatting this document. WBI Evaluation Studies are produced by the WBI Evaluation Group (WBIEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available online at: http://info.worldbank.org/etools/WBIEG/publications/index.cfm?pg=getPubs&category= Publications&Intro=yes&instructions=no&showDetails=no&ID= Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Manager Ms. Jaime B. Quizon ii ACRONYMS IEG Institute Evaluation Group WBI World Bank Institute WBIEG World Bank Institute Evaluation Group FY02-03 Fiscal year 2003-2004 FGDs Focused Group Discussions CRS Client Recording System DL Distance Learning NGOs Non-Government Organizations CBS Central Bureau of Statistics K & S Knowledge & Skill GoI Government of Indonesia CD Capacity Development TTL Task Team Leaders OLS Ordinary Least Squares UCM Unified Case Management 2SLS Two-Stage Least Squares iii TABLE OF CONTENTS ACKNOWLEDGEMENTS.................................................................................................ii ACRONYMS......................................................................................................................iii TABLE OF CONTENTS ...................................................................................................iv EXECUTIVE SUMMARY.................................................................................................v 1. INTRODUCTION ..........................................................................................................1 Background ................................................................................................................1 Evaluation Objectives and Questions.........................................................................1 Evaluation Methodology and Design.........................................................................2 Data Collection Methods ....................................................................................2 Evaluation Design and Sampling........................................................................3 2. RESULTS .......................................................................................................................5 Relevance of the Learning Events..............................................................................6 Activity Effectiveness................................................................................................7 Participants' Use of Awareness, Knowledge and Skills (K&S) from the WBI Activity.......................................................................................................................9 Follow-Up Activities with WBI Participants...........................................................12 Determinants of Activity Effectiveness and Participant Use of K&S .....................13 3. CONCLUSIONS AND RECOMMENDATIONS .......................................................20 Annex 1. Indonesia: Evaluation Questions and Data for Answering These Questions by Data Source ........................................................................................22 Annex 2: World Bank Institute (WBI) Country Focus Evaluation Questionnaire ..23 Annex 3: List of FY02-03 WBI Learning Events with Indonesian Participants .....29 iv EXECUTIVE SUMMARY This report evaluates WBI activities offered to Indonesian participants in FY02-03. These activities took place largely before the Institute started to implement a country- focused, capacity development (CD) strategy. This evaluation builds upon the IEG framework established earlier for similar evaluations for five WBI focus countries (Brazil, Egypt, Russia, Sri Lanka, and Thailand) completed in FY04. It relies mainly on participants' perceptions of the WBI training they received, although considers also the views of relevant Bank staff working on Indonesia. It uses mixed evaluation methods and data, namely: (a) quantitative survey data based on participant responses (N=114), and (b) qualitative data from two focus group discussions (FGD) with select participants and from interviews with Bank staff working on Indonesia. The objectives of this evaluation are: (a) to assess the relevance, effectiveness and impacts of past WBI learning events and (b) to establish useful benchmarks for future evaluations of WBI's upcoming activities for Indonesia under the Institute's new country-focused CD strategy. Related to these two objectives, the key findings of the this evaluation are: · Indonesian participants' average ratings of the relevance, effectiveness and impacts of the WBI learning events they attended are positive, but not extraordinary when compared with WBI benchmarks of 85 percent of training participants providing a rating of "4" or "5" on Level 1 evaluations of activity relevance, effectiveness and impact. Their use of the knowledge and skills that they acquired from the WBI activities is also positive, but unexceptional. Compared with similar non-WBI learning events offered in Indonesia, knowledgeable WBI alumni rate the usefulness of the WBI activity as only slightly better. · Our data from the survey of Indonesian WBI alumni, particularly the composition of this cohort of WBI participants and their ratings of the relevance, effectiveness and impact of the knowledge and skills they received, define some useful parameters against which future WBI Indonesian activities might be assessed. As noted already, there is significant room for WBI to improve on these initial indicators of its overall performance in Indonesia. Indeed, there are suggested ways already for improving on these performance indicators. FGD participants and survey results recommend: · the better selection of invitees to WBI learning events, recognizing the political and social contexts of institutional and policy changes in Indonesia; · the employment oflocal facilitators for DL courses; v · translation (if not, delivery) of courses in the language of proficiency of participants; · the use of work-related, action-learning activities, · an increase in the number and the focus of follow-up activities, particularly those that encourage further discussions on the most relevant topics of the learning event. WB resident staff suggest that WBI: · include resident staff more closely in the selection of the participants and the design, delivery and follow-up activities of WBI CD learning interventions; · tailor WBI activities also to the priority needs of the regions (i.e., subnational units) where capacity is severely lacking; · work closely with more local organizations as partners, including those at the propinsi and kabupaten levels, particularly for follow-up CD activities; · set clearly focused, more narrowly targeted goals that would allow impacts to be more evident and easily measured. WBI's own Client Recording System (CRS) includes client records for FY02-03 CRS that are severely incomplete. Reliable participant records are vital for WBI's country-focused agenda where systematic and sustained activities are required for capacity building. vi 1. INTRODUCTION BACKGROUND 1.1 This report forms part of a series of retrospective country-focused assessments of the relevance, effectiveness and impacts of past World Bank Institute (WBI) learning events. These evaluations encompass the training provided in FY02-03 to participants from seven WBI priority countries, namely: Burkina Faso, Guatemala, Indonesia, Kenya, Nigeria, Tajikistan, and Yemen. These evaluations, prepared by the WBI Evaluation Group (IEG), are of activities that took place largely before WBI started to implement a country-focused, capacity development strategy. It builds upon the IEG framework established earlier for similar evaluations for five WBI focus countries (Brazil, Egypt, Russia, Sri Lanka, and Thailand) completed in FY04.1 This evaluation report is for Indonesia. EVALUATION OBJECTIVES AND QUESTIONS 1.2 The objectives of these country-focused evaluations are: (a) to assess the outcomes and impacts of WBI learning activities in these countries; to form recommendations for improving ongoing WBI programs; and (b) to establish initial benchmarks against which the outcomes and impacts of future WBI country-focused interventions can be assessed. 1.3 This evaluation is meant to answer the following general questions: · Q1. What is the relevance of WBI learning activities to Indonesia's needs? · Q2. What is the effectiveness of WBI learning events?2 · Q3. What are the impacts of these WBI learning activities?3 · Q4. What features of WBI events are related to effectiveness and impact? 1.4 Annex 1 summarizes these questions in greater detail and the sources of data used to answer these questions. Additionally, the evaluation explores in-country factors, that may enable or hinder the effectiveness and impact of the WBI interventions. It also investigates whether or not WBI brings any unique and/or special elements into the Indonesia development context. 1These earlier reports are: Eckert, Sousa and Gunnarsson (2003) for Brazil; Zia (2003) for Egypt; Bardini, Manjieva, Narozhnaya and Gunnarsson (2003) for Russia; Khattri (2003) for Sri Lanka; and Quizon and Chard (2003) for Thailand. 2Effectiveness is defined in terms of raising participant awareness, increasing their knowledge, helping them develop strategies to address country needs, and so on. 3Impact is measured by the degree to which the activity influenced or led to changes in the areas of research, training, legislation and regulation, and country development strategies. It measures the contribution of WBI learning programs/activities to building in-country capacity. 1 1.5 In all, there were 70 FY02-03 WBI courses or learning activities (of more than one-day duration) that had Indonesian participants. At least 495 Indonesians attended these WBI learning events, which cover a variety of themes. Annex 3 lists the titles and duration of these 70 FY02-03 WBI courses. EVALUATION METHODOLOGY AND DESIGN Data Collection Methods 1.6 This evaluation involved three related information collection methods: a formal survey of FY02-03 WBI Indonesian participants, focused group discussions (FGDs) with Indonesian participants, and informal interviews with WB operations staff working on Indonesia. Key elements of these activities are described in Box 1.1. Box 1.1: Three Information Collection Methods Used Participant survey The survey collected information from randomly selected FY02-03 WBI Indonesian participants. This seven-page survey was first pre-tested on actual Indonesian participants before it was formally launched. An experienced Indonesian survey consultant and her staff administered the pre-tests and the actual survey. This survey, which collected, inter alia, participants' ratings of the relevance, effectiveness, and impact of WBI activities, is the primary data source used to answer the evaluation questions listed in Annex 1. Focus group discussions The aims of the FGDs were: to verify and elaborate on answers to evaluation questions from the survey; to obtain more information that may have been left out of the formal survey; and to assess the comprehensiveness of the coverage of the participant survey in getting at the relevance, effectiveness, and impacts of WBI activities. We conducted two FGDs with participants from two different WBI learning events. The first was held in Jakarta and had twelve participants from a learning event organized by WBI's Poverty and Growth thematic group. We hosted a second FGD in Bandung, West Java which had five participants from an event delivered by the City and Urban Management thematic group. Both FGDs lasted for about two and a half hours and involved a WBI facilitator and local Indonesian interpreters. WB Operations staff interviews We interviewed several WB operations staff working on Indonesia, both in Jakarta and in Washington, DC. Additionally, we distributed a simple questionnaire (see Annex 2) to higher-level WB staff working in Jakarta. This questionnaire basically asked about their awareness of FY02-03 WBI activities in Indonesia and their assessments of these WBI events, particularly in terms of their alignment with country priorities and their impacts, if any, on WBI participants, local institutions and government policy. 2 Evaluation Design and Sampling 1.7 We evaluate WBI FY02-03 learning events with longer than one-day duration, as impacts of anything shorter are likely to be negligible or non-existent. However, we include one-day events (or courses) that are part of a series of related activities but count the entire series as a single WBI learning event. We include only FY02-03 learning events because asking participants to recall anything beyond two years ago can be difficult and because the elapsed time between the WBI event and the survey should have been sufficient for any effects deriving from a WBI learning activity to have manifested themselves already.4 1.8 In sampling from Indonesian FY02-03 participants of WBI learning events for the survey, we first defined an eligible survey participant as one who had attended an eligible event, had at least one contact information (i.e., an e-mail address, mailing address, fax number, or telephone number) available in WBI's Client Recording System (CRS) database, are not staff members of the World Bank and are residents of Indonesia. Box 1.2: Sampling of Survey Respondents For the participant survey, there were 495 unique Indonesian names listed as FY02-03 participants in the CRS. Of these, we identified 417 as eligible participants. Mainly, this suggests that useful information exists for 84 percent of listed participants in the CRS. We randomly sampled 240 names from this eligible list for the participant survey.1 Of these 240 names, some 151 participants, or 63 percent, were contacted successfully for the survey. Of the 89 names that were not contacted, ten were purposely dropped because they were World Bank staff and not client participants (or the main target of this survey), and three names were double listed and were also dropped. The remaining 76 names, or about a third of the original sample, did not respond despite at least three separate attempts to contact them. Most of these names had wrong contact addresses, although a few could very well have simply ignored all of several attempts (by e-mail, phone or fax) at contacting them. 1.9 In the course of the actual survey, we commonly encountered wrong or outdated CRS-listed contact addresses (see Box 1.2). To illustrate, of the 151 sample respondents we were able to contact, 45 of them (or about 30 percent) had wrong or outdated CRS information but were tracked down successfully by our local Indonesian consultant with some effort. This difficulty with reaching past WBI graduates suggests that their greater likelihood of exclusion from any WBI follow-up or other activities stemming from the learning event. Clearly, WBI-wide participant data collection and updating efforts need to improve substantially, particularly in the context of "country focus" where the goal is capacity development through longer-term relationships with key local participants. In 4Also, a later intent is to combine participant survey data from the earlier five-country retrospective evaluations with very similar survey data collected for the current seven-country exercise, which includes Indonesia. Joining these data sets would be easier since the elapsed time allowed to measure impacts of a WBI activity will be comparable, and the potential recall bias the same, for both sets of country-focus evaluations. 3 order to ensure that WBI is able to maintain useful engagement with the majority of its alumni, incentive-driven WBI-wide efforts to improve the collection and updating of participant data might be helpful, if not necessary. 1.10 Of the 151 participants whom we reached, 114 eventually completed the survey. This amounts to a 48 percent response rate from the original randomly drawn sample of 240 names, or to a more encouraging 75 percent response rate from those participants whom we were able to reach. Our survey experience shows that participants, even high- level officials as evidenced from the breakdown of our final sample, are generally willing to fill in a questionnaire or be interviewed. The most difficult and time-consuming part of the survey is tracking down participants, particularly when their CRS-reported contact information is incorrect or incomplete. 4 2. RESULTS 2.1 FY02-03 WBI learning activities still cater largely to the needs of centrally located Indonesian authorities and institutes. Of the eligible survey participants listed in the CRS, 70 percent of them appear to be metro-Jakarta residents.5 This suggests that the main capacity development and other impacts of WBI interventions, if any, largely benefit the national capital center and its neighboring provincial/municipal areas. This is despite Bank-supported initiatives at decentralizing government -- including strengthening regional institutes and empowering regional authorities -- that started in 1997-98 following the national financial crisis. Because support of a decentralized administrative structure is a key element of the World Bank's country assistance strategy (CAS) for Indonesia, a more decentralized delivery of WBI learning events will not only align better with the CAS but also add to the relevance and effectiveness of these events. 2.2 The distribution of FY02-03 WBI participants, as noted in Table 1, shows that the majority were male (58 percent) and worked in government, i.e., either in national (30 percent) or provincial/local (29 percent) bureaucracies. Participants were about evenly distributed in terms of the level of their job positions with 39 percent of them reporting the highest or senior level positions (minister, full/associate professor, department head, senior researcher), 34 percent claiming middle level positions (program manager, project leader, assistant professor, technical expert), and the remaining 27 percent recording junior or entry level positions. Participants' primary work areas were in management or administration (34 percent), research or teaching (36 percent), and policymaking or legislation (29 percent). In all, with the exception of the Jakarta location of the majority of attendees, WBI learning events seem to attract a generally broad and balanced spectrum of Indonesian participants.6 Table 1. INDONESIA: Respondent Characteristics Participant Characteristic Distribution in % Residence Metro-Jakarta 70.2 Other 29.8 Gender Female 42.1 Male 57.9 Work Organization University/research institution 11.8 NGO, non-for-profit 13.8 Private sector 7.3 National/central government 30.5 Provincial government 12.2 Local/municipal government 17.3 Other 7.2 (Table 1 continues on next page.) 5See para. 1.8 for the definition of an eligible survey participant. 6We note that although some 30 percent of WBI participants are from provincial, local/municipal governments (see Table 1), the large number of these participants are from local governments within the metro-Jakarta area. 5 (Table 1 continued) Participant Characteristic Distribution in % Primary Type of Work Research 30.5 Policymaking/legislation 18.6 Management/adminstration 33.7 Teaching 5.8 Provision of services (e.g., financial, 1.3 health, etc.) Other 9.4 Level of Position Held Highest 9.3 Senior 29.8 Middle 34.3 Junior/entry 26.5 2.3 Participants' average proficiency in the language of instruction of FY02-03 WBI learning events is low, and this important issue needs to be addressed in future WBI Indonesia offerings. Only 44 percent of survey respondents, rated themselves as proficient -- i.e., they self-rated themselves a 6-7 on a 7-point scale -- in the language of instruction of the WBI event. A slightly higher 52 percent rated themselves as proficient in the technical terminology used in the learning event. Several participants in two separate focus group discussions (FGDs) raised the issue of having difficulties following the WBI course. In these FGDs, some participants also raised the question of selection of attendees. Because the main courses were delivered in English (with no local translators), a number of them were, in fact, nominated only because they were among the best English language speakers in their respective offices. Others in the same FGDs volunteered that even with that being the case, their English skills were not necessarily sufficient to follow all the material that was presented and discussed in these WBI learning events. Indeed, our conduct of the FGDs required local interpreters in both instances. This only demonstrates the language difficulties that these Indonesian FGD participants must have encountered during the actual WBI learning event. These FGD participants, who were mostly middle and high-level government officers, could not reiterate enough the importance of having a local translator (or simultaneous translation) or on-site facilitator for future WBI events, particularly for DL events such as theirs which were broadcast specifically to include a selected Indonesian audience. RELEVANCE OF THE LEARNING EVENTS 2.4 Participants gave high overall ratings for the relevance ofthe WBI learning events that they attended. On a 7-point scale (where, 1 = "not relevant" and 7 = "extremely relevant"), survey respondents rated the event's relevance to their own work and to Indonesia's needs as 5.4 and 5.8, respectively. Less than half (or 44 percent) of them noted that the WBI activity they attended was designed specifically for Indonesia. These estimates are useful benchmarks against which the impacts of upcoming learning event and other deliveries for Indonesia, under WBI's new country-focus strategy, might be measured.. 2.5 Relevance might also be gleaned from how respondents answered the question: Since the activity, have you discussed the issues raised in the activity at work, with local partners, government officials, NGOs or with the media? With options ranging from 1 = 6 "never discussed" to 7 = "thoroughly discussed", participants gave this question an average response of 5.0. In all, some 33 percent of respondents answered a 6 or 7 to this question, another 32 percent responded with a 5. 2.6 In the FGD discussions, participants agreed that the new materials discussed in their WBI events were relevant, although not all WBI materials were equally relevant to their jobs. Again, the question of participant selection emerged as a key issue here. For example, in one FGD on a poverty-themed WBI event conducted solely for participants from Indonesia's Central Bureau of Statistics (CBS), not only did the attendees have different levels of knowledge and skills --- characteristic of the difference in the levels of knowledge and skills between central and regional staff --- but the interests of attendees from the Jakarta-based CBS (mainly on policy studies and policy) were also very different from the concerns of participants from the regional CBS satellites (mainly on indicators and their measurement and other more basic poverty-related issues). 2.7 In summary, while Indonesian WBI graduates gave high overall ratings for the relevance of the learning events they attended, these ratings are not extraordinary; particularly when compared with the WBI benchmark of 85 percent of training participants providing a rating of "4" or "5" (on a 5-point scale) on Level 1 evaluations of activity relevance, effectiveness and impact. Also, these initial measures of activity relevance are useful benchmarks against which upcoming Indonesia-focused WBI interventions might be assessed. As noted, there is ample room for WBI to improve on these baseline scores under its new country-focus strategy. ACTIVITY EFFECTIVENESS 2.8 We asked respondents a series of questions to obtain some measure of the "effectiveness" of the learning event. We asked them to self-rate, again on a seven-point scale, the impacts of his/her participation in a specific WBI learning activity in six areas: raising their awareness and understanding of the development issues; providing them with the essential knowledge or skills; enhancing their understanding of their role as agents of change in Indonesia's development, providing them with approaches and strategies for addressing Indonesia's development needs; offering approaches for addressing the needs of their organization; and introducing respondents to others also interested in the theme of the learning activity.7 2.9 Figure 1 shows that respondents rated WBI activities above the mid-rating (of 4.0) in all six areas of effectiveness. In general, the loftier are the stated goals, the lower are participants' ratings of activity effectiveness. While actual range of these ratings vary from "not effective at all" (=1) to "extremely effective" (=7), on average, Indonesian participants rated WBI activities to be effective in raising individual awareness of Indonesia's development needs (=5.4); providing essential knowledge or skills (=5.4); enhancing understanding of their role in Indonesia's development (=5.2); offering approaches for addressing the needs of their organization (=5.0); providing approaches 7The response options ranged "not effective at all" (=1) to "extremely effective" (=7). "Not applicable" was also an option. 7 and strategies for addressing Indonesia's development needs (=4.8); and offering networking opportunities (=5.1). Figure 1. INDONESIA: Mean Rating of Effectiveness of the Activity, by Area of Effectiveness Providing knowledge or skills 5.5 5.4 Raising awareness/understanding 5.3 of dev't issues 5.2 Developing contacts/partnerships, 5.1 build coalitions in the field Rating 5.0 Helping understand role as agent of 4.9 change Mean 4.8 Developing strategies/approaches 4.7 for organizational needs 4.6 Developing strategies/approaches 4.5 for country needs Area of Effectiveness TOTAL Note: 7-point rating scale used: 1= "not effective at all"; 7 = "extremely effective" 2.10 Again, our FGD findings support these results, in general. The majority of participants in the poverty-focused FGD noted that the activity raised their awareness about the many dimensions of poverty while a similar majority in the urban-focused FGD cited their learning of new useful paradigms and concepts. Better networking opportunities were mentioned also as something that the WBI event provided. According to FGD participants, even though many of the attendees in each of these two WBI events belonged to the same institution or had met previously because of their shared job interests, the WBI event allowed them to establish familiarities that facilitated their meeting or communicating even after the event. 2.11 In order to obtain a general sense of the effectiveness of WBI activities, we created a summary gauge of effectiveness by averaging participants' scores across the individual effectiveness items. The average for this composite measure of effectiveness is 5.2. Also, some 38 percent of respondents rated WBI activities a 6-7 on this aggregate measure. We have no equivalent benchmarks with which to compare these Level 3-4 estimates of activity effectiveness, although in themselves, these survey-derived measures make useful baselines for assessing future WBI performance in Indonesia. When compared however with the WBI benchmark of 85 percent of training participants providing a rating of "4" or "5" (on a 5-poiny scale) on Level 1 evaluations of activity relevance, effectiveness and impact, these initial Level 3-4 estimates appear to be unsatisfactory.. 8 Figure 2. INDONESIA: Distribution of Respondents Scoring WBI Activity as "Highly Effective", by Area of Effectiveness 50% Raising 45% awareness/understanding 6-7 of dev't issues 40% Providing knowledge or 35% skills Scoring 30% Helping understand role as agent of change 25% 20% Developing strategies/approaches for 15% organizational needs Respondents 10% Developing strategies/approaches for 5% country needs Developing 0% contacts/partnerships, build Area of Effectiveness coalitions in the field PARTICIPANTS' USEOF AWARENESS, KNOWLEDGE AND SKILLS (K&S) FROM THE WBI ACTIVITY 2.12 Similar to their ratings of activity effectiveness, participants rated the change brought about by the WBI activity they attended as positive but not extraordinary. To the question: "how would you rate the change brought by the activity in the main topic or issue it addressed?", survey participants responded with an average rating of 5.3 (again based on a 7-point scale), with 37 percent of them responding with a 6-7. These figures are not statistically different from the aggregate effectiveness ratings noted earlier of 5.2 and 38 percent, respectively. 2.13 How did participants use the K&S that they gained from the WBI activity that they attended? Again, based on a 7-point scale where 1="not at all" and 7="very often"8 , the survey asked respondents to rate how often they had used what they had acquired from the WBI event in seven specific areas. With the actual range of these participant ratings varying from 1 to 7, on average, Indonesian respondents rated their use of K&S from the WBI activity as follows (see Figure 3): conducting research (=4.5); teaching (=4.6); raising public awareness in development issues (=4.7); implementing new practices within their work organization (=4.5); organizing collective initiatives (=4.6); influencing legislation and regulation (=4.2); implementing country development strategies (=4.2). These average ratings are significantly lower than similar average participant ratings for relevance and effectiveness noted earlier. While they indicate that participants find some use for the K&S that they obtain from attending the WBI event, their ratings appear to be hardly noteworthy. What might explain these results? 8"Not applicable" was also an option. 9 Figure 3. INDONESIA: Mean Ratings of Frequency of Use of K&S Obtained from the WBI Activity, by Purpose of Use 4.8 4.7 Conducting Reseacrh 4.6 Teaching 4.5 Frequency 4.4 Raising public awareness of Use 4.3 of Implementing new practices in 4.2 work organization Rating 4.1 Organizing collective 4.0 initiatives Influencing legislation and Mean 3.9 regulation 3.8 Implementing country development strategies Purpose of K&S Use Note: 7-point rating scale used: 1= "not at all"; 7 = "very often" 2.14 In the FGDs, participants noted several factors. Even if they wanted to implement what they had learned they are constrained by the structure of decision making in their organizations. The majority felt that the awareness of development issues of their own superiors should be raised in order for any institutional change to take root and flourish. Given the hierarchical structure of most Indonesian bureaucracies, the strong need for consensus at the top, and the importance of unambiguous guidance for lower level officials before anything new can happen, the careful choice of participants in focused WBI events is vital. Some FGD participants complained that participation in learning events, whether from WBI or elsewhere, are sometimes considered as ways of rewarding favored, but not necessarily the most qualified, employees, with overseas training programs the most highly coveted.9 The incentive systems in their offices are also a hindrance, particularly among government servants, because training is only marginally linked, if at all, to promotions and "standing out" with new ideas makes one only more vulnerable to criticism, particularly in public institutions where seniority and an egalitarian concept of fairness seem to matter more than performance.10 Consensus is very important in the Indonesia context and selection of participants towards this end is necessary for any desired impacts to happen more rapidly. Better participant selection is one challenge that future WBI country-focused interventions in Indonesia needs to address. 2.15 Finally, participants in the FGDs also mentioned the problem of work overload. There is only a thin layer of qualified high and middle echelon officials in many GoI 9This is particularly true of learning events held outside the participant's place of work (e.g., Jakarta) because of the travel and per diem benefits that these supposedly entail. In many cases, attendance in overseas training programs are reserved for very high-level bureaucrats. 10One FGD participant suggested that perhaps formal course credits by WBI (for some of its K&S courses) and recognition of these by the GoI might be a practical and effective incentive. 10 institutions, particularly in regional offices. On this account, these officials are usually overworked and overburdened with routine work that make it difficult to institute any change. The same FGD participants, however, also realize that many WBI learning events aim correctly to mitigate this undesirable situation. 2.16 As a summary measure of use of K&S acquired from the WBI event, we created a composite measure of "use" as the simple average of the seven specific areas listed in the survey questionnaire. This average aggregated measure of "use" is 4.5, with some 23 percent of respondents rating a 6-7 for this indicator. Figure 4 summarizes the proportion of participants who rated a 6-7 for each component of this "use" indicator. Figure 4. INDONESIA: Distribution of Respondents Scoring K&S Acquired from the WBI Activity as "Highly Used", by Purpose of Use 35.00% 7- Conducting Reseacrh 6 30.00% Teaching 25.00% Raising public awareness Scoring 20.00% 15.00% Implementing new practices in work organization 10.00% Organizing collective initiatives 5.00% Influencing legislation and Respondents regulation 0.00% Implementing country development Purpose of K&S Use strategies 2.17 The survey asked participants to rate the degree to which the activity influenced or led to changes in the same areas of use listed for the previous question. Based again on a 7-point scale, where "negative influence"=1; "neither helped nor hurt"=4; and "greatly helped"=7, survey respondents basically reported positive, albeit low ratings, as follows (see Figure 5):11 research (=5.6); teaching (=5.5); public awareness and development issues (=5.5); new practices within you work organization (=5.3); collective initiatives (=5.2); legislation and regulation (=5.2); and country development strategies (=5.3). The simple average for the aggregate of these components is 5.4, or just below the midpoint between "neither helped nor hurt" and "greatly helped". Although positive, these findings are not very comforting. They indicate largely mediocre results on the influence/impact of WBI learning events in Indonesia. 11Note that for this question, the neutral to positive ratings range from "neither helped nor hurt"=4 to "greatly helped"=7. Anything below a 4 is a negative rating. 11 Figure 5. INDONESIA: Mean Ratings of WBI Activity-Led Changes, by Area of Influence 5.7 Reseacrh 5.6 Teaching 5.5 5.4 Raising public awareness Ratings 5.3 New practices in work 5.2 organization Mean Collective initiatives 5.1 5.0 Legislation and regulation 4.9 Country development strategies Area of Influencia Note: 7-point rating scale used: 1= "negative influence"; 4 = "no influence"; 7 = "positive influence 2.18 Participants in the FGDs suggested the use of more action learning exercises tied directly to their work programs, including preparations by participants themselves before the training (e.g., request participants to bring certain problems they face into the learning activity). Application of newly acquired K&S through team-based exercises is also suggested, as for instance, to require participants to produce a plan (or targeted outputs) for public presentation --- to their regional bosses and other interested parties --- upon completion of the WBI course. Some WBI events already have these action-plan exercises. From the survey, 56 percent of respondents note that the WBI activity they attended had an action-learning components Of these respondents, 82 percent noted that they used part or all of the plan/strategy they developed in the WBI activity in their work. 2.19 Some 28 percent of survey respondents reported that they had participated in learning events offered by other organizations in Indonesia that were similar to the WBI activity they attended. These participants rated the usefulness of the WBI activity relative to the non-WBI event they attended (again on a 7-point scale where "much less useful"=1; "about the same"=4; and "much more useful"=7) a 4.6. Thus, WBI-sponsored events appear, on average, to be more useful to participants than those learning activities offered by others in Indonesia. This overall relative advantage for WBI, however, is not very heartening. FOLLOW-UP ACTIVITIESWITH WBI PARTICIPANTS 2.20 The difficulties with contacting past graduates of WBI events for this evaluation already attests to the problems with any WBI follow-up activities. Additionally, only 32 percent of respondents to the participant survey reported that WBI contacted them for follow-up issues regarding the activity. A much smaller 17 percent reported having contacted WBI themselves for follow-up issues or for questions on the content of the WBI activity. These low levels of follow-up efforts do not bode well for WBI's capacity development (CD) goals. 12 2.21 A large part of this problem arises because of the shortfalls in the coordination of WBI activities with Jakarta-based WB staff. For instance, among the seven higher-level staff members in the WB resident mission in Jakarta whom we interviewed or who responded to the questionnaires sent to them, only four could recall correctly FY02-03 WBI learning events with Indonesian participants -- mainly in their own areas of expertise -- while the other staff members were either too new to the Indonesia country office or could not recall any specific learning event with Indonesian participants. All, however, did not know or were not correctly aware of the number of events (even in their areas of expertise) that WBI offered which had Indonesian participants. These staff members raised as a key concern the lack of communication/coordination between them and the WBI TTLs who, at times, deal directly with Indonesian institutions and/or participants. Evidently, in order to have greater in-country impacts, future WBI country- focused CD initiatives should more closely involve resident mission staff in the selection, design, delivery and follow-up of these learning opportunities. Better WBI record keeping methods and follow-up efforts with local institutions/individuals are also vital because CD impacts require sustained and targeted efforts. DETERMINANTSOF ACTIVITY EFFECTIVENESS AND PARTICIPANT USEOF K&S 2.22 What factors explain the effectiveness of WBI learning events? What are the determinants of a participant's use of the K&S acquired from WBI events? We use a straightforward 2SLS regression model to answer these two questions, where the first stage initially explains what determines activity effectiveness. The second stage then relates these estimates of activity effectiveness and other variables to measures of use of the learning activity. 2.23 We first define an aggregate measure of effectiveness as a composite variable, i.e., the average of participant's ratings across the six areas of effectiveness, namely, effectiveness with: (a) raising individual awareness of Indonesia's development needs; (b) providing essential knowledge or skills; (c) enhancing understanding of their role in Indonesia's development; (d) offering approaches for addressing the needs of their organization; (e) providing approaches and strategies for addressing Indonesia's development needs; and (f) offering networking opportunities. 2.24 We note here that we also used decompositions of this aggregate measure to define more focused effectiveness indicators, namely: academic effectiveness (the average of items (a), (b) and (c) above); operational effectiveness (the average of items (d), (e) and (f) above). These component indicators, however, did not fare any better as dependent variables in first-stage regressions compared with our single aggregate measure of effectiveness. 13 2.25 We also define separate measures of a participant's use of WBI-acquired K&S from the activity. The first is aggregate use: a composite variable that is the average of a participant's ratings across the seven areas of possible use, namely, use in: (a) conducting research; (b) teaching; (c) raising public awareness in development issues; (d) implementing new practices within their work organization; (e) organizing collective initiatives; (f) influencing legislation and regulation; and (g) implementing country development strategies. 2.26 We also decomposed this aggregate measure of use into the following components: academic use (the average of items (a), (b) and (c) above); operational use (the average of the last four items above). 2.27 We explored several variables that might explain activity effectiveness and use. All these independent variables, as explained in Box 2.1. are all from the participant survey, unless otherwise noted. Box 2.1.: Regression Variables Activity-level variables: · DL = 1, if the activity attended was delivered via video sessions; · duration = actual number of days of the learning activity (from CRS data); · designed for country = 1, if learning activity was thought by respondent to be designed for Indonesian participants; · program type: urban = 1, if the participant participated in a program sponsored by WBI's Urban and City Management thematic group (from CRS data); · program type: poverty = 1, if the participant participated in a program sponsored by WBI's Poverty and Economic Management thematic group (from CRS data); · activity location = 1, if learning activity was held in Indonesia (from CRS data); · action plan = 1, if participant said he/she developed an action plan/strategy during the activity to apply the K&S learned; · percent in-country participants = actual percent of participants from the country attending the learning event (from CRS data); · contact information = 1, if participant was provided contact information for other participants; · WBI follow-up = 1, if WBI contacted participant after the learning event; · participant follow-up = 1, if participant contacted WBI regarding the activity after the learning event. (Box 2.1. continues on next page.) 14 (Box 2.1. continued.) Participant-level variables: · gender = 1, if female; · proficiency in language of instruction = participant's self-rated score on a 7- point scale of his/her proficiency in the main language of instruction of the activity; · organization: university/research = 1, if participant worked the longest in a university or research institution since the learning activity; · organization: government = 1, if participant worked the longest in national, provincial or local government since the learning activity; · job position =1, if participant worked at a middle-level position (e.g., program manager, project leader, assistant professor, technical expert); = 2, if participant worked at the highest (e.g., minister, deputy minister, full professor, president of an organization) or senior-level (e.g., department head, division head, associate professor, senior researcher) position the longest since the learning activity; · job type: research/teaching = 1, if participant worked the longest in research or teaching since the learning activity; · job type: policy making/legislation = 1, if participant worked the longest in policy making or legislation since the learning activity; · job type: management/administration = 1, if participant worked the longest in management or administration since the learning activity; · job type: services = 1, if participant worked the longest in the provision of services (e.g,, financial, health, and so on) since the learning activity. Exogenous variables: · work environment = actual participant rating on a 7-point scale of the degree to which work environment factors (e.g., work procedures, colleagues, incentive system, resources, and so on) hurt/help use of K&S acquired form the learning activity; · country development environment = actual participant rating on a 7-point scale of the degree to which country-level development factors (e.g., country policies, social groups, political groups, readiness for reform, and so on) hurt/help use of K&S acquired form the learning activity; · discussions after learning activity = actual participant self-rating on a 7-point scale, where 1=never discussed and 7=thoroughly discussed, of his/her discussion of the issues raised in the learning activity at work; with local partners, government officials, NGOs; or in the media. 2.28 We tested several factors that might explain our composite measure of activity effectiveness, including activity related variables, participant-related characteristics and exogenous factors. Table 2 reports our regression estimates for aggregate effectiveness. Among all the OLS models that we tested, these two regression estimates 15 have about the highest explanatory power (i.e., R2 = .18 and .21, respectively).12 It also includes all those explanatory variables that appear to be the most consistently significant and meaningful in the other test regressions.13 2.29 What features of WBI events and participants are related to effectiveness? As shown, there are not many. Four key variables have positive and significant effects on effectiveness.: (a) development of an action plan, (b) whether or not the participants attended a learning event organized by the WBI"s Urban and City Management (or UCM) thematic group (or program type: urban),14 (c) the participant's proficiency in the language of instruction, and (d) whether or not the participant worked in a university or research institution since the learning activity (or organization: university/research). 2.30 The first two of these four significant explanatory variables are activity-level variables; they suggest what activity features WBI might aim for in order to raise its participants' ratings of activity effectiveness. Action plan development (or action learning) is a notable WBI activity feature that improves overall effectiveness. Also, that one WBI thematic program (i.e., UCM) stands out as being quite effective suggests that in the context of improving overall WBI effectiveness in Indonesia, there may be lessons to be learned from a closer scrutiny of the learning activities of this specific thematic group. 2.31 The last two significant variables are participant-level variables; they suggest the types of participants who are most likely to rate the activity they attended as highly effective. These participants appear to be those with some proficiency in the language of instruction of the learning event and those who are affiliated with a university or research organization. For WBI then, the challenges are to match the language of delivery of the learning event to that in which the participant is proficient, and to broaden the appeal of its learning events to comprise others beyond local researchers and academics. 12We also tested several regressions with decomposed measures of activity effectiveness as the dependent variable. These alternative estimates did not show any significant improvements over the result reported in this Table 2. 13We also tested OLS regressions with activity relevance as explanatory variables, i.e., where the variable "relevance to work" = participant's rating on a 7-point scale of the degree to which the learning event was relevant to the participant's work rating, and the variable "relevance to country" = participant's rating on a 7-point scale of the degree to which the learning event was relevant to Indonesia's needs. Because they are highly corrected with measures of effectiveness, both variables were highly significant in all OLS results. Their inclusion in the estimated regressions made all other dependent variables statistically irrelevant. 14The omitted category for this "program type" variable is program type: other. 16 Table 2: Activity Effectiveness Regression Estimates Activity effectiveness aggregate aggregate Designed for country 0.271 0.306 [1.40] [1.55] Action plan 0.406 0.339 [2.11]** [1.73]* Proficiency in language of instruction 0.144 0.107 [1.67]* [1.19] DL 0.264 0.199 [0.98] [0.73] Percent In-Country Participants -0.009 -0.009 [1.18] [1.23] Activity Location -0.022 0.019 [0.04] [0.04] Program type: urban 0.76 0.668 [1.99]** [1.71]* Program type: poverty 0.466 0.353 [1.62] [1.14] Organization: University/Research 0.628 [1.76]* Organization: Government 0.239 [0.90] Constant 4.151 4.216 [8.14]*** [7.81]*** Observations 93 93 R-squared 0.18 0.21 Absolute value of t statistics in brackets * significant at 10%; ** significant at 5%; *** significant at 1% Table 3: Activity Use: Second-Stage Regression Estimates Activity Use Aggregate Academic Operational Activity effectiveness: aggregate 0.766 1.018 0.841 [2.19]** [2.06]** [2.00]* Work environment 0.012 0.013 -0.068 [0.11] [0.08] [0.52] Country development environment 0.007 0.05 0.08 [0.06] [0.30] [0.65] Discussions after learning activity 0.289 0.139 0.244 [2.41]** [0.80] [1.73]* Action plan 0.11 -0.227 0.278 [0.43] [0.63] [0.98] Contact information -0.35 -0.557 0.037 [1.34] [1.53] [0.13] WBI follow -up -0.193 -0.025 -0.166 [0.59] [0.05] [0.47] Participant follow -up 0.202 0.639 0.085 [0.47] [1.07] [0.18] (Table 3 continues on next page.) 17 (Table 3 continued.) Activity Use Aggregate Academic Operational Organization: university/research -0.092 -0.362 0.221 [0.17] [0.49] [0.39] Organization: government -0.049 -0.238 -0.164 [0.16] [0.55] [0.47] Job type: research/teaching -0.069 0.605 -0.666 [0.19] [1.18] [1.70]* Job type: policy making/legislation 0.122 0.332 0.154 [0.28] [0.54] [0.32] Job type: management/admiistration -0.425 -0.314 -0.441 [1.14] [0.61] [1.10] Job type: services -0.5 0.157 -0.765 [0.68] [0.15] [0.96] job position 0.763 0.354 0.831 [1.70]* [0.56] [1.71]* Gender -0.258 -0.091 -0.324 [1.14] [0.29] [1.31] Constant -0.837 -1.353 -1.185 [0.60] [0.69] [0.73] Observations 73 72 71 R-squared 0.5 0.12 0.55 Absolute value of t statistics in brackets * significant at 10%; ** significant at 5%; *** significant at 1% First stage equation: q5_tot=q3 q16 q21 orguniv orggov dl partratio loc_id pr_urb pr_pov 2.32 Table 3 reports second-stage regression estimates explaining activity use. The first estimate explains aggregate use while the last two report the second-stage regression results for academic use and operational use, respectively. In all three regressions, predicted effectiveness from the second equation in Table 2 is used as the explanatory variable. As expected, effectiveness is a positive and significant predictor of aggregate use, and its components: academic use and operational use. However, there are not many other variables that significantly explain use. 2.33 Whether or not the participant had discussions after learning activity of issues raised in the activity is a positive and significant factor explaining operational use but not academic use. The same is true for position level, where participants in higher positions are more likely to report higher and significant operational use, but not academic use. This latter result only supports an earlier concern raised in the FGDs that in the Indonesia context, organizational hierarchies matter for effecting any changes from the learning activity. For WBI, therefore, these results suggest that for Indonesia, the operational use of the K&S that its graduates acquire may be raised by preferably selecting participants who are in high-level positions and by strongly encouraging WBI alumni to continue to discuss after the event the issues raised during the learning activity. 18 2.34 Finally, job type: research/teaching has a negative and significant effect on operational use, i.e., participants who claim to be in the research/teaching professions are likely to report significantly lower levels of operational use than job type: others professions.15 This finding is not too surprising given that academics are not as likely to use the K&S they acquired to actively implement, influence, or organize for institutional changes. 15We note here that the variable job type: others is the omitted category in these regressions. Job type: others refers to all other jobs that cannot be classified as research or teaching, policy making or legislation, management or administration, or services. 19 3. CONCLUSIONS AND RECOMMENDATIONS 3.1 Participants' average ratings of the relevance, effectiveness and impacts of the WBI learning events they attended are positive, albeit not extraordinary, particularly when compared with the WBI benchmark of 85 percent of training participants providing a rating of "4" or "5" (on a 5-point scale) on Level 1 evaluations of activity relevance, effectiveness and impact. Their use of the K&S that they acquired from the WBI activities is also positive, but unexceptional. Compared with similar non-WBI learning events offered in Indonesia, knowledgeable WBI alumni rate the usefulness of the WBI activity as better, but unremarkable. 3.2 The survey and other data collected for this study offer very useful parameters for gauging the relevance, effectiveness and impacts of upcoming WBI activities for Indonesia under the Institute's country-focused capacity development strategy. And, as noted above, there exists significant room for WBI to improve on these initial benchmark indicators of its overall performance in Indonesia. 3.3 There are ways for improving on these performance indicators already. FGD participants and survey results recommend: · the better selection of invitees to WBI learning events. FGD participants commented that selection of Indonesian participants to WBI events should recognize how institutional and policy changes are made in different Indonesian institutions that are targeted for change including inter alia how government-sponsored training opportunities are oftentimes used for non- training purposes -- for instance, as favors/rewards to certain officials. · translation (if not, delivery) of courses in the language of proficiency of participants. Our survey results suggest that selection of participants who are proficient in the language of instruction of the WBI event will raise ratings of effectiveness and use. This criteria might also be used more strictly when selecting participants to WBI events. · the use of more work-related, action-learning activities. FGD participants suggest the employment of local facilitators for DL courses for these action-learning activities. While this might be an additional expense for course delivery, this extra effort should tie in very well with getting useful follow- up CD activities off and running. · an increase in the number and the focus of follow-up activities. Of particular interest to WBI graduates are accessible activities that encourage further open discussions on the most relevant topics of the learning event. But follow-up activities are constrained naturally by available resources on hand. There are the familiar tradeoffs here between broadening vs. deepening the 20 effectiveness and impact of WBI-supported events, for which there are no clear answers as yet. 3.4 WB resident staff suggest that WBI: · include them more closely in the selection of the participants and the design, delivery and follow-up activities of WBI CD learning interventions; · tailor its activities also to the priority needs of the regions (i.e., outside Jakarta) where capacity is severely lacking;16 · work closely with more local organizations as partners, including those at the regional level, particularly for follow-up CD activities; · set clearly focused, more narrowly targeted goals that would allow impacts to be more evident and easily measured. 3.4 Finally, it is important to improve WBI's Client Recording System (CRS). Client records should be more up to date. Reliable participant records are vital for WBI's CD country-focused agenda where systematic follow-up activities are required for institution building. 16Also, some WB staff suggested that WBI define more clearly what being a "priority country" means in the context of Indonesia. 21 ANNEX 1. INDONESIA: EVALUATIONQUESTIONS ANDDATA FOR ANSWERING THESEQUESTIONSBY DATASOURCE Evaluation Question Source I. What is the relevance of WBI learning activities to Indonesia's needs? To what degree are the topics covered by WBI learning activities relevant to Indonesia's Participant Surveys & specific needs? Interviews w/ Country Staff Do WBI learning activities address Indonesia's current issues and needs? Participant Surveys & Interviews w/ Country Staff II. What is the effectiveness and impact of WBI learning activities?* How effective are WBI learning events in Indonesia? (Effectiveness is defined in terms Participant Surveys of three components: knowledge and skills, strategies and approaches and networking). To what degree are the participants utilizing the knowledge/skills they learned through Participant Surveys & WBI learning activities/products? And in what areas? How useful is the activity? Interviews with Country Staff What are the Indonesia-specific facilitators and barriers to utilizing the knowledge/skills? Participant Surveys & Interviews w/ Country Staff Has the activity influenced or contributed to changes in the relevant sector/area? Participant Surveys & Interviews w/ Country Staff III. What is the contribution of WBI learning programs/activities to building in-country capacity, relative to others delivered by other members of the development community? To what degree do WBI programs build sustainable in-country capacities in learning and Participant Surveys & applying new knowledge and skills? [Country-specific indicators of sustainability include local partnerships, follow -up meetings mobilized by participants] Interviews w/ Country Staff What is the extent of similar learning activities provided by non-WBI organizations? How Participant Surveys & do their effectiveness and impact compare to WBI training programs? Interviews w/ Country Staff 22 ANNEX 2: WORLDBANK INSTITUTE (WBI) COUNTRYFOCUSEVALUATION QUESTIONNAIRE Instructions WBI had the pleasure to have you participate in the following learning activity: Title: _____________________________________________________________ Held from: ________________________ to: ________________________ In: _______________________________________________________________ Getting your opinion of the above-mentioned activity--now that you have had time to reflect on it--is very important to help WBI improve its programs. For this, we ask you to complete this questionnaire. The questionnaire has four sections and should take approximately 20 minutes to complete. · Section 1 asks about the relevance of the activity. · Section 2 asks about the usefulness of the activity. · Section 3 asks you to compare this activity with similar learning activities offered by other organizations. · Section 4 asks about the characteristics of the activity, its follow-up and your background. We need your honest feedback. Please keep in mind that your responses will be kept confidential, and will be used for the sole purpose of improving WBI programs. If you have any questions about the questionnaire please call or send a message to Mr. Jaime B. Quizon at jquizon@worldbank.org . Thank you for agreeing to complete this questionnaire! ID:_________________ 23 World Bank Institute (WBI) Country Focus Evaluation Questionnaire I. Relevance of the Activity 1. Since the e nd of the activity, to what degree has the activity been relevant to yourwork? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 2. To what degree have the topics covered in the activity been relevant to your country's needs? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 3. Was the activity designed specifically for participants from your country? s Yes s No s Don't now 4. Was the activity related to the country development goals listed below? a. Eradicate extreme poverty Don't now s Yes s No s b. Achieve universal primary education Don't now s Yes s No s c. Promote gender equality and empower women Don't now s Yes s No s d. Reduce child mortality Don't now s Yes s No s e. Improve maternal health Don't now s Yes s No s f. Combat HIV/AIDS, malaria, and other diseases Don't now s Yes s No s g. Ensure environmental sustainability Don't now s Yes s No s h. Develop global partnerships for development Don't now s Yes s No s i. Ensure water sanitation and supply Don't now s Yes s No s j. Improve investment climate and finance Don't now s Yes s No s k. Promote trade Don't now s Yes s No s 24 II. Usefulness of the Activity 5. Please rate the degree ofeffectiveness of the activity in each area noted below. (If the area was not an objective of the activity, please mark "not applicable.") Not Areas Extremely Not effective at all effective applicable 1 2 3 4 5 6 7 NA a. Raising your awareness and understanding of the development issues important to your country s s s s s s s s b. Providing you with knowledge or skills s s s s s s s s c. Helping you better understand your role as an agent of change in your country's development s s s s s s s s d. Helping you develop strategies or approaches to address the needs of your organization s s s s s s s s e. Helping you develop strategies or approaches to address the needs of your country s s s s s s s s f. Helping you develop contacts, develop partnerships and build coalitions in the field s s s s s s s s 6. How would you rate the change --brought by the activity--in the main topic or issue it addressed? Strong negative No Change Strong positive Don't change change Know 1 2 3 4 5 6 7 DK s s s s s s s s 7. How often have you usedthe knowledge and skills you acquired in the activity for the following purposes? (If you have not worked in the given area since this activity, please mark "Not applicable.") Purposes Not Very Not at all often applicable 1 2 3 4 5 6 7 NA a. Conducting research s s s s s s s s b. Teaching s s s s s s s s c. Raising public awareness in development issues s s s s s s s s d. Implementing new practices within your work organization s s s s s s s s e. Organizing collective initiatives s s s s s s s s f. Influencing legislation and regulation s s s s s s s s g. Implementing country development strategies s s s s s s s s 8. To what extent did the following factorshelp or hurt the process of using the knowledge/skills that you acquired at the activity? Factors Greatly Neither helped Greatly Not hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA a. Your work environment (e.g., work procedures, colleagues, incentive s s s s s s s s system, funding, etc.) Factors Greatly Neither helped Greatly Not hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA b. Your county's development environment (e.g., country policies, social groups, political groups, readiness for reform, etc.) s s s s s s s s 25 9. How has the activity influenced or led to changes in the following areas? (If the area is not relevant to the activity, please mark "Not applicable.") Areas Negative No Positive Not influence influence influence applicable 1 2 3 4 5 6 7 NA a. Research s s s s s s s s b. Teaching s s s s s s s s c. Public awareness in development s s s s s s s s issues d. New practices within your work s s s s s s s s organization e. Collective initiatives s s s s s s s s f. Legislation and regulation s s s s s s s s g. Country development s s s s s s s s strategies 10. Since the activity, have you discussed the issues raised in the activity, at work, with local partners, government officials, NGOs, or in the media? Never Thoroughly discussed discussed 1 2 3 4 5 6 7 s s s s s s s III. Comparison of the WBI Activity with Similar Activities Offered by Other Organizations 11. Did you participate in any similar learning activities offered by other (NON-WBI) organizations in your country? (If no, please skip to question 14.) s Yes s No 12. If yes, please provide the name(s) of the organization(s): 1. 2. 3. 13. How would you rate the usefulness of the WBI activity compared to NON-WBI activities? WBI About the same WBI No much much opinion less more useful useful 1 2 3 4 5 6 7 s s s s s s s s 26 V. Characteristics of the WBI Activity, its Follow-up and Your Background 14. How would you describe thetype of the WBI learning activity that you attended? Mix of Video Video Sessions and (Distance Class room (Face Face to Study Learning) to Face) Face Conference Web-based Learning tour 1 2 3 4 5 6 s s s s s s 15. How effective was thistype of learning activity in helping you learn? Not effective at No all Extremely effective opinion 1 2 3 4 5 6 7 s s s s s s s s 16. During the WBI activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, or policy documents) to apply the knowledge and skills you learned? (If no, please mark "no" below, then skip to question 18.) s Yes s No 17. If yes, did you use part or all of the action plan in your work? s Yes s No 18. Were you provided with thecontact information of other participants in the activity, such as e-mail addresses, telephone numbers or mailing addresses? (If no, please mark "no" below, then skip to question 20.) s Yes s No 19. If yes, how did you use it? Used it to Used it to organize continue activity joint related follow-up Other uses Never used it discussions activities (Please specify briefly) s s s s ________________________________________ 20. Was the language of instruction used during the activity the same language you use at work? s Yes s No 21. At the time of the activity, what was your level of proficiency in the language of instruction? Not proficient at all Highly proficient 1 2 3 4 5 6 7 s s s s s s s 22. At the time of the activity, what was your level of proficiency in thetechnical terminology used in the activity? Not proficient at all Highly proficient 1 2 3 4 5 6 7 s s s s s s s 23. After the activity, did WBI contact you for follow-up issues regarding the activity? s Yes s No 24 After the activity, did YOU contact WBI for follow-up issues or questions on the content of the activity? (If no, please skip to question #23) s Yes s No 27 25. If yes, please rate WBI's helpfulness in addressing your issues. WBI responded, I did not have WBI did but wasnot follow-up not helpful WBI responded and was requests for respond at all extremely helpful WBI 0 1 2 3 4 5 6 7 NA s s s s s s s s s 26. Which of the following best describes the organization in which you have workedthe longest since the activity? (Select one.) s University/research institution s National/central government s Non-governmental organization(not-for- s profit) Provincial/regional government s Media s Local/municipal government s Private sector s Other, specify: ____________________________ 27. Which of the following best describes the primary type of work you have donethe longest since the activity? (Select one.) s Research s Teaching s Policymaking/legislation s Provisionofservices(e.g.,financial,health,etc) s Management/administration s Other, specify: ____________________________ 28. How would you best describe the level of the position you have held the longest since the activity? s Highest level (e.g., Minister, Deputy Minister, Top Government Official, Full Professor, President of an organization) s Senior level (e.g., Department Head, Division Head, Associate Professor, Senior Researcher) s Middle level (e.g., Program Manager, Project Leader, Assistant Professor, Technical Expert) s Junior Level (e.g., Research associate, Ph.D. level graduate student, Technical Specialist) s Entry level (e.g., Intern, assistant) s Other, Please specify: __________________________________________________________________ 29. What is your gender? s Male s Female Thank you for your feedback. We appreciate very much your cooperation. 28 ANNEX 3: LIST OF FY02-03 WBI LEARNING EVENTS WITH INDONESIAN PARTICIPANTS Session Title Start Date End Date AFDP Workshop on Developing Government Bond Markets in APEC Economies 12-Nov-02 15-Nov-02 Adapting to Change: Core Course on Population- Reproductive Health and Health Sector 19-Aug-02 30-Aug-02 Reform Advanced Impact Evaluation 6-Nov-02 11-Nov-02 Advanced Training in Fiscal decentralization in Developing and Transition Countries 23-Jul-01 3-Aug-01 Annual Financial Markets and Development Conference - Brookings Institute- International 14-Apr-03 16-Apr-03 Monetary Fund- and World Bank Asia Meeting on Efficient Clean Development Mechanism Operations 1-Apr-03 3-Apr-03 Asia Regional Air Quality Training Consortium Stakeholder Meeting-Clean Air Asia 28-May-02 31-May-02 Basic Poverty Measurement and Diagnostics 10-Jun-02 21-Jun-02 Basic PovertyMeasurement & Diagnostics Course 12-May-03 17-May-03 Business Ethics and Corporate Accountability: The Search for Standards 9-Sep-02 27-Sep-02 CAI-Asia Clean Air Regional Workshop and Second Coordination Meeting in Hanoi 5-Sep-01 7-Sep-01 CATNet-Asia: Training of Trainers for Basic Urban Air Quality Management Distance 16-Jun-03 19-Jun-03 Learning Course CATNet-Asia: Training of Trainers on Air Pollution Inspection Procedures and Safety 3-Mar-03 13-Mar-03 City Strategies to Reduce Urban Poverty through Local Economic Development 19-Nov-01 14-Dec-01 Clean Air Initiative Asia Regional Workshop: Household Energy- Indoor Air Pollution and 9-May-02 10-May-02 Health Core Course- Labor Market Policies 23-Apr-01 4-May-01 Corporate Governance and Strategy in East Asia 23-Apr-02 25-Jun-02 EAPA DLC Workshop - Learner Support and Delivery 23-Sep-02 28-Sep-02 Education: National Assessments in Asia 21-Apr-02 26-Apr-02 Enhancing Training Quality Through Customer Service (QCS-1): Caring for Clients 5-Jun-02 11-Jun-02 Before- During and After Training Environmental Economics & Development Policy 16-Jul-01 27-Jul-01 Financial Markets and Development Conference: In Whom We Trust 17-Apr-02 19-Apr-02 Flagship Core Course on Health Sector Reform and Sustainable Financing 13-Jan-03 31-Jan-03 Forest Law Enforcement and Governance 11-Sep-01 13-Sep-01 Frontiers in Infrastructure Finance 18-Jun-01 29-Jun-01 Global E-Conference on Business- Democracy- and Peace 7-Oct-02 1-Nov-02 Global E-Conference on CSR- Education and Technology: How Can Corporations 18-Nov-02 7-Dec-02 Contribute to Bridging Educational and Digital Divides? Global E-Conference on The Role Of Business In Fighting HIV/AIDS 21-Apr-03 9-May-03 Global Organization of Parliamentarians Against Corruption 13-Oct-02 16-Oct-02 Human Capacity Development for an Effective Response to HIV/AIDS 4-Jul-02 5-Jul-02 ICGFM-WBI Workshop on Reducing Poverty through Improving Public Expenditure and 25-Nov-02 27-Nov-02 Financial Accountability India: Water Sector Reform Capacity Building Initiative 18-Sep-01 19-Sep-01 International Conference on Environmental Compliance and Enforcement (INECE) 15-Apr-02 20-Apr-02 Introduction to Corporate Social Responsibility for Future Leaders 11-Jul-02 23-Jul-02 Irrigation Institutional Reforms Initiative 21-Apr-02 26-Apr-02 Leadership Program on AIDS: Distance Learning Series for East Asia: Cross-Border 1-Dec-01 1-Jun-02 Transmission and Related Issues- VC and Elec. Dis. Forum Management-focused Monitoring and Evaluation -MFM&E- 4-Jun-03 18-Jun-03 National Assessment of Educational Achievement: The Logistics of National Assessment 16-Dec-02 17-Dec-02 National Education Assessment: Development of pilot tests questionnaires 9-Dec-02 16-Dec-02 National Educational Assessment: Sampling 23-Sep-02 27-Sep-02 National Strategy Study (NSS) Workshop on Capacity Building Needs under the Kyoto 23-Sep-02 25-Sep-02 Protocol - Switzerland (Annex 3 continues on next page.) 29 (Annex 3 continued.) Session Title Start Date End Date New Technologies for Small and Medium-Size Enterprise Finance 4-Dec-02 6-Dec-02 Overview of Corporate Social Responsibility - 3: Business Ethics 5-Aug-02 9-Aug-02 Overview of Corporate Social Responsibility and Sustainable Competitiveness- EOA 26-Feb-03 25-Mar-03 PRSP for Lao- Vietnam- and Cambodia 10-Oct-02 15-Dec-02 Policy Primer on Microfinance: what have we learned ? 15-Oct-01 30-Oct-01 Promoting Practical Environmental Compliance and Enforcement Approaches in East Asia 9-Dec-02 12-Dec-02 II Promoting Practical Environmental Compliance and Enforcement- Approaches in East 24-Jun-02 28-Jun-02 Asia Regional Forum - Hanoi 5-Dec-01 7-Dec-01 Regional Forum on Effective Legislative Oversight 10-Dec-01 11-Dec-01 Regional Seminar for Asian Managers of Monitoring and Evaluation of Poverty Reduction 9-Oct-02 11-Oct-02 Programs Regional Workshps on Nonbanking Financial Institutions in East Asia - World Bank and 4-Sep-02 6-Sep-02 Securities and Exchange Commission in Thailand Reproductive Health and Health Sector Reform Course for UNFPA 28-Oct-02 1-Nov-02 Risk Management Workshop for Regulators: Assessing- Managing- and Supervising 27-Apr-03 2-May-03 Financial Risk South Asia Urban & City Management Course 14-Jan-02 24-Jan-02 Strategic Management and Marketing of Training (SMMT-5): Ensuring Sustainability and 22-Apr-02 3-May-02 Financial Health of Training Institutions The 3rd Clean Air Regional Workshop in Asia 7-Nov-01 9-Nov-01 The 4th Trainers' Workshop on "Improving Training Quality through Interactive Learning 4-Mar-02 20-Mar-02 Technologies and Distance Mentoring (ITQ-4)" The Americas E-Conference on Corporate Social Responsibility 10-Jun-02 5-Jul-02 The Challenge of Leadership 30-Mar-03 2-Apr-03 Third Annual Conference of the Parliamentary Network on the World Bank 9-May-02 11-May-02 Training Workshop on Project Formulation for the Kyoto Protocol s Clean Development 30-Sep-02 2-Oct-02 Mechanism (Vietnam) Training of Trainers Course on Basic Poverty Analysis (C101) 18-Feb-02 1-Mar-02 Urban Poverty Learning Workshop for Asia 10-Jun-02 11-Jun-02 Using Knowledge for Development 21-Nov-02 22-Nov-02 WATER-Course on Participatory Irrigation Management 3-Sep-01 13-Sep-01 WTO Coursefor Trade Policy Officials 6-Nov-01 7-Nov-01 Water Forum 2002 6-May-02 8-May-02 Workshop on Intergovernmental Fiscal Relations in East Asia 10-Jan-02 11-Jan-02 Youth and Corporate Citizenship: Future Leader's Forum 13-Nov-01 17-Nov-01 30