Y SST EMSAPPROAC O HF RBETE TREDUCAT IONREU S T LS 79902 Wok ring S P A a p B e r E S er R e is Number6Api rl2013 WhatMa ersfor Wok rfoc reD e velopment: AFa rmew or k andTool o frAnay lss i Contents Acknowledgements....................................................................................................................................... 3 Executive Summary ...................................................................................................................................... 4 Introduction ................................................................................................................................................. 5 Purpose and Audience ................................................................................................................................. 6 Relation to Other Skills-Related Work ......................................................................................................... 6 A Note on Terminology ................................................................................................................................ 8 Conceptual Framework .............................................................................................................................. 10 Linking Policies to System Performance .................................................................................................... 15 Data for Benchmarking WfD Policies and Institutions ............................................................................... 18 Overall Structure of the SABER-WfD Data Collection Instrument .......................................................... 18 Dimension 1: Strategic Framework ......................................................................................................... 20 Dimension 2: System Oversight .............................................................................................................. 24 Dimension 3: Service Delivery................................................................................................................. 29 Design Features of the SABER-WfD Data Collection Instrument ............................................................ 33 Analyzing the SABER-WfD dataset .............................................................................................................. 34 Conclusion ................................................................................................................................................... 39 References ................................................................................................................................................. 40 Appendix 1: Functional Dimensions, Policy Goals, and Actions in SABER-WfD .......................................... 49 Appendix 2: Examples of Benchmarking Rubrics for Topic-Level Data ...................................................... 50 Appendix 3: Composite Rubrics for the Policy Actions .............................................................................. 52 Appendix 4: Composite Rubrics for the Policy Goals .................................................................................. 62 2 Acknowledgements This paper has benefited from the support of the SABER-Workforce Development team, whose current core members include Rita D. Costa, Ryan Flynn, Rijak Grover, Brent Parton, and Viviana Gomez. The authors wish to thank them for contributing to the technical content of this paper, particularly the construction of the data collection instrument and the specification of the scoring rubrics. We gratefully acknowledge the collaboration of Robert McGough on the initial version of this paper, and also the insight shared by users of the pilot version of the SABER-WfD tool which helped improve the clarity of our conceptual framework and the precision of the data collection instrument. These users include Jutta Franz, Hye Won Ko, Susan Leigh-Doyle, Yoon Hee Park, Arwen Raddon and Cecilia Zanetta. We wish to thank numerous colleagues at the World Bank, including Omar Arias, Mamta Murthi, and Sukdeep Brar among others, as well as members of two advisory panels for helpful suggestions on the ideas presented at early stages of our project. 1 We are most grateful for financial support from the government of the United Kingdom through the Partnership for Education Development (PFED) between the World Bank and the UK Department for International Development (DfID); and from the government of the Republic of Korea through the Korean Trust Fund. None of the above individuals or parties is responsible for the views expressed in the paper or for any remaining errors; the authors alone are. Readers are welcome to send comments on the paper to the authors through Jee-Peng Tan at jtan@worldbank.org. 1The panels brought together individuals with relevant experience in education and training and in workforce development for discussion with the SABER-WfD team. One panel comprising retired World Bank staff met face-to-face with us on January 20, 2011; its members were: Arvil Van Adams (chair), Birger Fredriksen, John Middleton, Peter Moock, Juan Prawda, Michelle Riboud, Shashi Shrivastava, and Saint William. The other panel is made up of experts around the world who met with us through two staggered video conferences on October 26, 2010; it included the following individuals: David Ablett, Kenneth King, Song Seng Law, Young-Hyun Lee, Wong Fook Low, Peliwe Lolwana, Bianca Rohrbach, and Phillip Toner); and Hernan Araneda, David Ashton, Michael Axmann, Dan Baffour-Awuah, Renato Caporali, Gordon Nixon, Charlene Nunley, and Dan Vogler. 3 Executive Summary Equipping the workforce with job-relevant skills is a continuing challenge around the world. In developing and emerging economies, mismatches in skills demand and supply are a persistent concern. Many school leavers and graduates are unable to find jobs commensurate with their education and training while employers complain of difficulty in filling their vacancies. Employers also bemoan the scarcity of “soft skills� which are considered essential for business productivity in today’s global economy. More broadly, skills constraints are making it difficult for companies to innovate and move into more lucrative areas of economic activity. The inability to integrate into and ascend the value chain in global production networks keeps countries in a low-skills trap and impedes their efforts to accelerate economic growth and poverty reduction. The persistence of these problems suggests that workforce development in most low- and middle-income countries is not functioning as well as might be desired. The World Bank seeks to enhance its support to partner countries in addressing challenges in workforce development, in part through the creation of new analytical tools to inform policy dialogue. This paper elaborates on the analytical framework and practical design of one such tool: the Systems Approach for Better Education Results (SABER) tool for assessing workforce development (WfD) systems. The tool is part of a whole suite of SABER diagnostic tools created following the May 2011 launch of the World Bank’s Education Strategy 2020. These SABER tools share a common purpose: to enable systematic documentation and assessment of the policy and institutional factors that influence the performance of education and training systems. The focus of SABER-WfD is on how well the system is equipping individuals to meet the demand for skills in the labor market. The SABER-WfD tool places explicit focus on three functional dimensions of WfD policies and institutions: (a) strategy; (b) system oversight; and (c) service delivery. Strategy refers to the alignment between workforce development and a country’s national goals for economic and social development. System oversight refers to the governance arrangements that shape the behaviors key stakeholders involved, including individuals, employers, and training providers. Service delivery refers to arrangements for managing the provision of services in order to achieve results on the ground. The tool’s pilot testing in five countries--Chile, South Korea, Ireland, Singapore, and Uganda—suggest that the three broad functional dimensions are indeed the right ones to focus on in workforce development. As the tool’s application expands beyond the pilot countries, it is hoped that the comparative data thus generated would create a new knowledge asset on promising policies to advance workforce development in the World Bank’s partner countries. 4 Introduction This paper presents a diagnostic tool for assessing a country’s institutions, policies and praxis for workforce development (WfD). The tool is part of a broader suite of analytical resources associated with a new World Bank initiative called the Systems Approach for Better Education Results (SABER) which covers multiple policy domains, among them WfD (World Bank 2012a). 2 An experimental version of the SABER-WfD tool was tested in five countries (Chile, Ireland, Singapore, South Korea, and Uganda) and lessons from the pilot phase have helped to improve the tool. The purpose of this paper is to explain the motivation for creating the tool and to offer insights into its analytical and operational design. 3 A tool such as SABER-WfD is especially appropriate at this juncture in the World Bank’s work. Many of the institution’s partner countries have invested in education and training and have made good progress in expanding coverage at all levels. They are also paying more attention to learning outcomes, to the broader array of skills beyond academic achievement, and to technical and vocational education and training. 4 Yet, a growing pool of better educated and qualified labor has not always produced the expected results. Many countries experience the paradox of having large numbers of graduates that are unable to find jobs commensurate with their education and training (or indeed any job at all) while employers complain of skills shortages and mismatches. These challenges highlight the need for a re- examination of the age-old issue of skills supply and demand. The World Bank’s 2013 World Development Report (2013) offers a comprehensive perspective on jobs, including the relation between skills and jobs. For the purpose of this paper, suffice it to note that rapid technological progress is changing not only the types of jobs available but also their content. Autor et al. (2003), for example, document the role of information and communication technology in shifting the pattern of work in the U.S. in favor of jobs requiring higher order skills at the expense of those needing only repetitive manual skills. The influence of technology on jobs is not limited to advanced countries. By weaving countries more tightly into the global economy, technology is exposing them all to opportunities for new sources of growth as well as to threats to the status quo (Memedovic 2004; Milliron 2007). These trends are part of what Schumpeter (2003) call the process of “creative destruction� which increases the likelihood of economic restructuring as firms push into new areas of businesses with higher returns while abandoning activities for which they are no longer competitive in the global marketplace. 5 An increasingly inter-connected global economy has implications for all countries. Countries with youthful populations face the daunting task of equipping large numbers of first-time job seekers with the skills needed to make a smooth transition from school to work. To realize their ambition for economic growth and development, these countries must insert their industries into global value chains 2 The SABER initiative is elaborated in the next section. 3 The SABER-WfD tool comprises a conceptual framework and an accompanying data collection instrument (DCI) that generates qualitative data for assessing countries’ institutions, policies and praxis for workforce development. The DCI is currently in version 2.5 and is available upon request to the authors. Tan et al. (2012) contains an explanation of the pilot version of the analytical framework and data collection instrument. 4 One reflection of the increasing focus on learning outcome s is the prevalence of international projects to measure learning outcomes. Examples include Trends in International Mathematics and Science Study (TIMSS), Program for International Student Assessment (PISA), and Programme for the International Assessment of Adult Competencies (PIAAC). 5 See Nadvi (2012) for a fascinating account of the pressures of globalization and technological change on the soccer ball manufacturing industry in Pakistan. 5 and production networks (Memedovic 2004). This means equipping their workforce with the business and technical capabilities required to produce and sell more in global markets (World Bank 2003). In other countries, population aging is the more important problem. As part of reforms to manage the fiscal viability of pension systems, the mandatory retirement age is likely to rise even as economic restructuring increases the prospect of multiple job transitions for incumbent workers. The combination of these factors implies that life-long learning will grow in relevance and importance as working adults seek new or better skills to remain employable throughout their (longer) working lives. The growing interest in workforce development thus provides an important motivation for the creation of the SABER-WfD tool. The rest of the paper contains six sections. The two sections immediately below elaborate on the purpose and intended audience of this new diagnostic tool; and situate it in relation to other resources for skills-related work at the World Bank. The section that follows clarifies two terms used throughout this paper: workforce development (WfD) and benchmarking. The next four sections present the paper’s substantive core: the conceptual framework, key propositions on the link between policies and institutions and outcomes in workforce development, the design of the data collection instrument and the strategy for data collection and analysis. The last section concludes the paper. Purpose and Audience As a member of the development community, the World Bank assists partner countries to advance socioeconomic progress, including in the area of workforce development. 6 Its involvement takes many forms, which for simplicity may be categorized under three broad spheres of activity: knowledge generation, policy dialogue, and operational support. For country-specific engagement, these categories are often combined to provide customized packages of services and financial support. This paper aims to enhance the first two categories of assistance. To elaborate, the paper presents and explains the design of a new diagnostic tool for systematic collection and analysis of data on key functional dimensions of workforce development policies and institutions. The tool is intended to shed light on an important part of the “black box� that separates tangible inputs from outcomes. It seeks to help bridge the gap between analysis of problems in workforce development—as captured in various indicators of outcomes or performance—and the solutions required for improvement. While the search for solutions has been a staple of policy dialogue, the tool provides all who engage in the process with a more systematic way to assess institutional impediments to better outcomes in workforce development. Furthermore, the tool’s application in multiple countries is expected to culminate in a comparative database of good practices that can be mobilized easily to foster cross-country exchange and learning. 7 Relation to Other Skills-Related Work The new tool fills a niche in the broad range of World Bank resources available to enhance its support for workforce development in partner countries. Besides the numerous regional and country-specific 6 The World Bank’s partner countries refer to low- and middle-income countries as well as fragile states. 7Lessons from the pilot phase have informed the specification of protocols for data collection and analysis. Adherence to these protocols is helping to enhance the comparability of cross-country qualitative data. 6 studies, these resources include four major conceptual resources that are noteworthy. 8 The most recent of these is the World Bank’s 2013 World Development Report on Jobs (World Bank 2012b). While the Report addresses broad issues about jobs beyond workforce development, it highlights several key ideas relevant to the concerns of this paper. The Report reiterates the importance of literacy, numeracy, as well as social skills for jobs; it notes that job opportunities increase the demand for education and skills building; it highlights that having a job enables people to learn and build skills; and it draws attention to the fact that the availability of skills can attract jobs, particularly where countries are undergoing rapid urbanization and integration into the global economy. The second conceptual resource is the World Bank’s Skills toward Employability and Productivity (STEP)9 framework. Created in 2010, it highlights a holistic model for considering the role of human development in supporting socioeconomic progress. The model encompasses five components: (a) starting right in early childhood; (b) laying a strong foundation in basic and secondary education; (c) building and upgrading job-relevant skills; (d) fostering innovation and entrepreneurship; and (e) matching skills demand and supply. Workforce development focuses on—but is not strictly limited to— the third of these five components. A third conceptual resource is the 2012 “The Right Skills for the Job?� study which elaborates on the STEP framework. It updates our understanding of the sources of market and government failures in equipping workers with job-relevant skills through initial and continuing education and training, as well as targeted services to vulnerable populations through active labor market programs. The book recognizes that in today’s modern economy innovation and entrepreneurship are two critical competences required to boost individual and firm productivity. The important role of market intermediation to match skills demand and supply also receives attention in the book. Completing the set of conceptual resources now available is the World Bank’s Education Strategy 2020 which was published in May 2011 following approval by its Board of Governors. 10 The document reaffirms the institution’s mission in the education sector of supporting partner countries to invest in people’s knowledge and skills to promote development. To implement the Strategy, the Bank is prioritizing two specific aspects of its work: strengthening systems to deliver better results and building a knowledge base to inform policy and project design. Accordingly, it launched the Systems Approach for Better Education Results (SABER) Initiative to systematize the documentation and analysis of countries’ policies and institutions in education and training (Rogers 2013 forthcoming). Workforce development (WfD) is among the policy domains addressed in the SABER Initiative. 11 The creation of the SABER-WfD tool is consistent with the growing emphasis on measurement and evidence in the World Bank’s work on skills. With regard to measurement, recent studies have pointed to the inadequacy of common indicators such as educational attainment and related details on fields of study, type of training, and so forth (see, for example, Hanushek and Wößmann 2006; Sondergaard and Murthi 2012). 12 These indicators also neglect the “soft� skills valued by employers (see, for example, di 8 As of this writing, regional flagship reports on skill-related themes have been completed in five of the World Bank’s administrative regions, while the report for the Africa Region is underway as of this writing. See XXX to access these reports. 9 For more details on the STEP framework, see World Bank 2010a. 10 See World Bank (2011) and http://go.worldbank.org/DTQZ9EKJW0 (accessed on November 9, 2011). 11 See World Bank (2012) and the resources at http://go.worldbank.org/NK2EK7MKV0 (accessed on November 9, 2011). 12 While useful, these indicators are insufficient for policy development, given their generally weak link to individuals’ labor market outcomes and to countries’ rate of economic growth. 7 Gropello et al. 2010 and 2011) and rewarded by market forces (see, for example, Heckman et al. 2006). To narrow the measurement gap, the World Bank is currently implementing a multi-country project known as the Skills for Employment and Productivity (STEP) Skills Measurement Study. 13 This survey is expected to add to our understanding of the nature of skills gaps and mismatches and to motivate in- depth policy dialogue. Complementing the STEP Skills Measurement study, the SABER-WfD tool is expected to enhance such dialogue through its systematic documentation and analysis of likely institutional barriers to better outcomes. 14 With regard to strengthening the evidence base of the World Bank’s work on skills, the tool bears a more tangential relationship: it seeks merely to document the practice of impact evaluation in a given setting rather than delve into the details of such evaluations. Rigorous impact evaluation is a separate line of work at the World Bank that is now gaining increased attention in light of significant knowledge gaps about what does and does not work in skills development (e.g., Betcherman et al. 2004 and 2007).15 On a more limited and selective basis, the World Bank also fosters South-South exchanges in order to mobilize evidence from the perspectives of practitioners. These include study visits that enable policy makers in partner countries to gain first-hand exposure to and engage in dialogue and exchange with their peers in other countries. A Note on Terminology Workforce development and benchmarking are two terms that are used frequently in this paper. Because they bear different meanings to different people, it is useful to clarify at the outset how we use them in the content of the SABER-WfD project. Workforce development has attracted growing interest in recent years from a wide range of researchers and policy makers, many of whom are concerned with the implications for skills development associated with five major factors: globalization, technology, the role of knowledge and innovation in the “new economy,� political change, and demographic shifts (Jacobs and Hawley 2008). Although the term raises issues at the level of individuals, organizations, and societies, we use it in this paper to refer to the challenges faced by a national, regional, provincial, or sector-based system in responding to two expectations: • Enabling individuals to acquire the knowledge, practical skills, and attitudes for gainful employment (including self-employment) or improved work performance in a particular trade or occupation; and • Providing employers with an effective means to communicate and meet their demand for skills. 16 13 Results from the study are expected in 2013. 14 Several of the countries (e.g., Laos and Sri Lanka) that are implementing the STEP measurement study are thus also engaged in a parallel study using the SABER-WfD tool. 15 For more information on ongoing evaluations of skills-related initiatives, see www.worldbank.org/sief (accessed on November 9, 2011. 16 This dual expectation is shared by the U.S. National Governors Association, which notes that to develop the skilled and knowledgeable workforce required for economic competitiveness in the global economy, “state workforce and education systems are designed to provide the skilled workers employers need to thrive and the education and training individuals need to prosper in today's labor market.� See http://www.nga.org/cms/home/nga-center-for-best-practices/center-issues/page-ehsw- issues/page-workforce-development.html (accessed on August 12, 2012). 8 With regard to the first expectation, it is clear that job-relevant skills are acquired through diverse pathways: some are established in early childhood education (for example, language skills) and reinforced through subsequent general schooling at the primary and secondary levels (for example, literacy and numeracy); others rely on pre-employment vocational training at the secondary and post- secondary levels; and still others through apprenticeships, on-the-job training via formal and non-formal arrangements, and informal learning. A complete evaluation of workforce development thus requires a comprehensive assessment of the entire education and training system and its effectiveness with regard to the skills of both incumbent workers and the future pipeline of workers. Such a comprehensive assessment is beyond the scope of the SABER-WfD project. Instead we narrow our focus to those parts of the system with an explicit mission of enabling individuals to acquire job-relevant skills, either through initial preparation or continuing education and training. The tool thus places little focus on early childhood education, primary education, and general secondary schooling, even though these are all clearly essential in laying a strong foundation for skills acquisition. To avoid overlap with the SABER tools addressing higher education, SABER-WfD focuses on the segment of the workforce filling mid-level positions (for example, skilled craftsmen, technicians, production supervisors, and so forth). Formal training in the skills associated with such positions typically begins at the secondary or post-secondary non-university levels and can progress up to the tertiary level in some fields. 17 The second expectation of workforce development puts the spotlight on employers as users of skills. As such, their involvement in WfD is essential to ensure a responsive system for skills supply. In a well- functioning WfD system, employers are able to meet their demand for skills and compete successfully in product and service markets while individuals are able to acquire the knowledge and skills to prosper in the labor market. Such a system can help boost economic productivity and growth; creating and maintaining its performance is thus a key concern of policy makers pursuing skills-intensive strategies for economic development. Benchmarking is another term used in this paper. The concept gained prominence in 1979 in light of what Xerox Corporation did to stem the loss of its photocopier business to Japanese competitors (Camp 1989). The company undertook a systematic assessment of its business practices, in the process identifying ways to improve manufacturing design, boost production efficiency, and lower costs, all of which enabled it to regain its competitive edge. The concept and practice of benchmarking have since become widespread as a well-regarded and versatile tool for evaluating organizational and institutional performance (Dattakumar and Jagadeesh 2003). Often used as a continuous process for enhancing outcomes, benchmarking now finds wide application in the public and private sectors throughout the developed world. 18 An organization that embarks on benchmarking assesses various aspects of its processes in relation to those of leading organizations, usually from a peer group defined for the purposes at hand. The exercise poses such questions as the following for self-examination: (a) how do we compare to others in the same business; (b) how good do we want to be; (c) who is doing it the best; (d) how do they do it; (e) how can we learn from and adapt what they do to suit our circumstances and needs; and (f) how can we excel and become a standard of 17 University-level training typically supplies workers with advanced technical and research skills that are essential for fostering the development of knowledge-intensive economic sectors. It is the focus of a separate domain in the SABER project and is therefore examined more fully there. 18 See, for example, http://www.well.com/user/benchmar/germany/Germany.html (accessed on November 9, 2011). 9 performance for others to aspire to? 19 The process allows an organization to develop plans for adapting specific best practices to improve its own performance. For our purpose, national or subnational systems of WfD rather than companies are the relevant units of analysis. 20 Conceptually, the foregoing questions relating to the benchmarking of companies remain relevant when considering systems of WfD. Many countries are interested in knowing how well their policies, institutions and praxis compare to those in nations with more effective systems for workforce development. To systematize the comparison, we develop performance metrics for key aspects of WfD policies and institutions and define the corresponding stages of development based, whenever possible, on the experiences of nations with successful WfD systems. Where data are scarce, the benchmarks are based on a review of the relevant literature and on the logic of practical progression as a system improves. Finally, it is important to emphasize that the purpose of benchmarking is not to rank countries but to create a common frame of reference for policy discussion and to facilitate self-diagnosis as a basis for improvement. Conceptual Framework Figure 1 illustrates the key elements of the conceptual framework for the SABER-WfD project. 21 The framework recognizes that achieving coherence in WfD policies requires a simultaneous consideration of the demand for skills as well as the supply. A well-functioning WfD system with a high degree of coherence between skills demand and supply achieves faster economic growth by: (a) improving trainees’ employability; (b) supplying the workers needed by firms to improve productivity, product quality, and competitiveness; and (c) enabling firms and entrepreneurs to expand their technological capability and move up value chains where profit margins are often more attractive. Where this match is poor, slower growth and the associated problems of joblessness and underemployment, brain drain and technological stagnation or laggardness are an ever-present prospect. 19 Adapted from Alstete (1995). 20 See O’Lawrence (2007) for an application of the benchmarking approach to a system for skills development in the state of California in the United States. Although the approach may be used to evaluate specific programs for skills development (for example, adult worker retraining and re-employment schemes, formal industry-based apprenticeship schemes, or disadvantaged youth employment/training schemes), the SABER-WfD project will focus on national or subnational systems. 21 The proposed benchmarking approach complements the World Bank’s MILES framework, which is used by some to evaluate employment issues through five lenses: Macroeconomic conditions, Investment climate and infrastructure; Labor market regulation and institutions; Education and skills development; and Social protection, where the capitalized letters are used to form the acronym MILES (see Banerji et al. 2008). 10 Figure 1: Conceptual Framework for Workforce Development Source: Authors’ construction. In developed countries, the economic crisis of 2008 and its aftermath have increased the urgency of addressing WfD issues (see, for example, OECD 2010a). In less-developed countries, skills to increase the employability of youth and to help accelerate overall growth also feature regularly in both national and international dialogue (see, for example, World Bank 2010a; World Bank 2012b). Below we elaborate on the main components of the framework relating to skills demand and supply and to the linkages between these two parts of a WfD system. The demand for skills. The government’s economic policies set the overall context for skills demand in multiple ways. Wage policies and related legislation, for example, influence the flexibility of the labor market, the hiring practices of firms, and the aggregate demand for labor and its distribution between formal and informal sectors. 22 Tax policies affect firm and worker behavior, including their incentives to train. The spending policies of governments often have a significant impact on skills demand. Large infrastructure investments, for example, create a demand for various construction, engineering, and specialized skills. In some countries, governments also make strategic investments to nurture or grow selected sectors and these may call for new skills beyond what is available in the current labor pool (see, for example, UKCES 2009). In countries where the government takes a proactive approach to economic management, its policies may also affect firms’ business strategies and the incentives for innovation and adoption of new technologies. These forces in turn shape the demand for skills that may be required for technology absorption and advancement. 22In Mexico, for example, Levy (2008) notes that the government requires firms to provide health insurance, pensions, and other social benefits directly to salaried workers, whereas non-salaried workers get these services through government pension and insurance programs. The difference in policy effectively puts a tax on salaried workers but offers a subsidy to their peers in the informal arrangements. Levy estimates that salaried benefits in Mexico are 30–35 percent less than their actual value to workers; and that the gap is sufficiently large to keep workers in low-productivity work in the informal sector, thus slowing overall economic growth. 11 Forward-looking policies must obviously take into account global trends and practices and evaluate their implications for the local economy. Demographic changes, globalization and technological advances, urbanization, and climate change are some of the key influences on the dynamics of skills demand. The world’s population is aging as a result of declining fertility and rising life expectancy. 23 In developed economies, the trend is already translating into a growing demand for health and personal care services and for related skilled workers, often imported from abroad. 24 In less-developed economies, the share of youth in the labor force—many of them new entrants and often in the midst of starting their families—is increasing rapidly. They represent both a pool of potential employees and a market for consumer goods and services and the associated demand for skilled workers to produce and sell these products. In most countries, however, the potential has yet to translate into tangible results in the form of jobs and sustainable income growth. Globalization and technology, particularly in information and communication, are an even more obvious influence on the demand for skills. Firms everywhere face increasing pressures to innovate and diversify into newer and more lucrative markets, as profit margins in traditional industries are squeezed by intense global competition and the shortening of product shelf life. Among the more dynamic firms that compete in export markets, skills bottlenecks are a chronic complaint, even in countries with highly educated workforces. 25 Rapid urbanization, a trend that some countries actively promote in order to foster dynamic clusters of economic activity, is also ratcheting up the demand for talent and skilled workers, including service workers to build and maintain lively and livable urban spaces. In some settings, the lure of overseas jobs has fueled the demand for skills, encouraging potential migrants to obtain training to improve their prospects abroad. Climate change is yet another global trend with important implications for skills demand. As countries scale up mitigation or coping responses, they will require workers skilled in a wide range of “green� technologies. These global trends combine with conditions in a particular sector or a locality to shape the demand for specific skills. In the United States, for example, a 2009 survey of manufacturing businesses found emerging skills shortages in three sectors (aerospace and defense, life sciences and medical devices, and energy and resources) and highlighted bottlenecks for skilled production workers, and for scientists and engineers (DMIO 2009). In Europe, WfD policies are being informed by detailed analyses conducted by the European Center for the Development of Vocational Training for the 27 countries of the European Union. 26 In India, the boom in information technology (IT) and IT-enabled services is increasing the demand for technicians—estimated at 500,000 a year—as well as for English language skills (Sudan et al. 2010). One careful study shows that each new call center raises enrollments in English-language schools in the center’s vicinity by nearly six percent (Oster and Millett 2010). 23 In developed economies, 33 percent of the population in 2050 is projected to be at least 60 years old, compared with just 22 percent in 2010. In developing economies, the corresponding percentages are 20 percent and 9 percent. Africa has the youngest population, but even there, the share of the population 60 years and above is projected to rise from 5 percent in 2010 to 11 percent in 2050 (UN 2009). 24 At the same time, older workers in developed economies are finding it necessary to postpone their retirement and seek new skills in an effort to remain employable and productive. As a result the concept of lifelong learning has gained growing relevance in public policy debates. 25 See, for example, data from the World Bank’s Enterprise Surveys at http://enterprisesurveys.org. 26 According to CEDEFOP (2010), projections for 2020 suggest a net loss of some 2.5 million jobs in the primary sector (mainly agriculture) and 2.0 million in traditional manufacturing, balanced against gains of some 7.0 million jobs in business and other services, 3.4 million in distribution, transport, hotels and catering, and 1.0 million in education, health, and other non-marketed services. Demand is expected to rise for highly- or medium-skilled workers, while it will shrink for workers in clerical services, crafts, and related trades and agriculture. 12 The demand for quality skills. A large number of surveys confirm that employers value not just a worker’s ability to perform specific technical tasks, but also his or her possession of “soft� skills or behaviors (Ferrier et al. 2003a; UKCES 2009). In the United States, manufacturers have consistently ranked highly skilled and flexible workers as one of two top drivers of performance (the other is new product innovation). 27 Employer surveys conducted in 2008 in the Philippines and Indonesia revealed a high demand for three “soft� skills for managers and production staff alike: problem-solving, leadership, and communication skills. In Vietnam, the top behavior prioritized by employers in a 2009 survey was punctuality. The same survey also highlighted that employers valued workers’ practical and technical knowledge more highly than theoretical knowledge. A recent survey of Indian employers highlighted deficiencies in the “soft� skills of the graduates from engineering colleges (Blom and Saeki 2011). The importance placed on “soft� skills is not surprising given the nature of work in today’s economy. An increasingly integrated global economy with intense competition, a growing body of scientific and technological knowledge, and widening access to knowledge via digital media are reinforcing the demand for skills that go beyond simple book learning and/or the competence to execute only specific tasks. The ability to solve problems, to learn new skills, and to adapt to new conditions, all of which require a solid foundation in literacy and numeracy, are especially valuable in contexts of rapidly evolving business conditions (Wang 2012). 28 These profound changes are transforming today’s jobs, increasing the premium on workers that can perform non-routine tasks as a normal part of their jobs (Autor, et al. 2003; Autor 2010; Levy and Murnane 2005). In settings where most school leavers must find their livelihood in the informal sector, having the flexibility, entrepreneurship, and initiative to find sufficiently productive work must also be added to the list of core skills that young people will require for a successful transition from school to work (for example, Adams 2007). The supply of skills. The technical and vocational education and training (TVET) system is frequently perceived as the main source of workforce skills development. 29 While the perception is understandable in light of the government’s often significant role in providing TVET, it often does not reflect reality. In many, particularly developing countries, private formal and informal TVET are also major sources of TVET (Janjua and Naveed 2009; King and Palmer 2010). In middle- to high-income nations, and especially in large industrialized nations such as the United States, Germany, and Japan, employer-based training programs are the main suppliers of middle- to higher-level skills training. Because such training activities and programs are fragmented across the sector, typically funded by and often unpublicized outside the firms, their national contribution to WfD is not always well-understood or recognized. The wide range of sources for skills acquisition creates a complex landscape. The various formal, nonformal, and informal programs may each require different levels of prior educational attainment and may operate under different institutional homes, typically though not exclusively, the ministries of 27 See DMIO 2009. 28 Because of the importance of these cognitive skills, the World Bank recently embarked on the STEP Skills Measurement project, which involves administering tests to individuals aged 15–64 as part of the effort to document the levels and distribution of such skills in the workforce and to relate the possession of these skills to individuals’ earnings and employability (among other outcomes). 29 It is useful to note that the acronym TVET represents the first letter of four words with specific meanings: “technical� and “vocational� typically refer, respectively, to technician-level and craftsmen-level programs; “education� refers to exposure to the world of work and preparation for further studies in technical and vocational fields; and “training� refers to preparation for entry into, or upgrading in, specific occupations or clusters of occupations. 13 education and of labor. While both formal and nonformal TVET refer to structured programs, they often fall under the mandate of different parent ministries. Formal programs usually take place in the education system and end with formal certification or qualifications; nonformal programs often occur outside the education system (hosted by such ministries as labor, social welfare, industry, and so forth), in or outside the workplace, and may or may not lead to an accredited qualification. Where granted, the qualification may not relate to the certification system hosted by the education ministry. Informal TVET comprises unorganized and unstructured learning that is seldom the responsibility of any single agency; it often occurs in the workplace, for example, when a junior worker learns the ropes simply by working alongside or shadowing more experienced colleagues. 30 Despite their informality, these arrangements can be a significant source of learning for tacit skills; their prevalence often depends on firms’ workplace practices and organizational culture. To add to the complexity, the system for skills provision may also include such programs as: (a) formal, industry-based apprenticeships, usually for training in advanced skills; (b) training partnerships involving organized labor and firms, often with formal curricula taught by training providers; (c) technical training for and within the military; and (d) special needs training (that is, skills training for select populations, such as prisoners, persons with disabilities, and the disadvantaged). In all TVET systems, government policies influence to varying degrees the way the system is organized, governed, and financed and how skills attainment is certified and recognized. They therefore affect the efficiency of service provision, the effectiveness of the system in producing quality and relevant skills, and the accessibility of training opportunities, particularly among disadvantaged population groups. They also shape the incentives for both training providers and learners, and influence the aggregate supply of skills and its diversity by level and specialization. The impact of policies on the role of employers in WfD planning and implementation—at both the strategic and operational levels—warrants close attention. In well-performing systems, policies are in place to improve the responsiveness of training providers by encouraging them to view employers, both public and private, as critical partners whose input and feedback help ensure the continued relevance of training services against evolving labor market conditions. Such systems are demand-led (Middleton et al. 1993; Elkins et al. 2012). It is important to clarify that being demand-led does not always mean that the system is driven by employers alone. For countries trying to move into higher value-added production, politicians’ policies regarding the economy’s strategic direction also influence the demand for skills. A demand-led system would be sensitive to such demand as well. By contrast, in systems where the relation between training providers and employers is weak, the risk of mismatches in both the quantity and quality skills is ever present. In such supply-driven systems, employers that face critical skills gaps may resort to addressing the problem themselves, as in India in such sectors as information technology and communication and pharmaceuticals (as documented, for example, in Tan and Nam 2012). Alignment of skills supply and demand. A key challenge in all WfD systems is to sustain a good match between skills demand and supply even as conditions in the labor market evolve and alter the demand for skills. Where the match is good, significant benefits can accrue in the form of a dynamic and more productive workforce, higher rates of employment and labor utilization, progress in the fight against poverty and tangible movement up the value chain of economic activity for the economy as a whole 30Informal TVET refers to a method of training and should not be confused with TVET programs that are intended to equip trainees for jobs in the informal sector of the economy. 14 (see, for example, Ferrier et al. 2003b; Gereffi et al., 2011). The alternative scenario is that of a weak system delivering a poor match between skills demand and supply. In such systems, the risks include high unemployment and underemployment, often coexisting with chronic skills gaps felt by employers, emigration of skilled workers, and an economy showing few signs of diversifying and upgrading of its technological capabilities. In reality, skills misalignment is a common problem in most countries. Numerous surveys of employers consistently reveal concerns about skills constraints on business performance and about the lack of certain types of skills among employees (for example, di Gropello et al. 2010 and 2011). These problems stem to a large extent from what economists describe as market and government failures. Below we elaborate on an analytical strategy to benchmark the policies that governments might use to overcome such flaws and enhance outcomes in workforce development. Linking Policies to System Performance While we focus on government actions to promote a closer connection between skills demand and supply, this emphasis does not mean that the government is the most important actor among those with a role in WfD. Rather, it is intended to facilitate dialogue on how the government might be more effective, whether acting directly or through the intermediation of other actors, in advancing its goals for workforce development. Below we begin by elaborating on three broad areas where weaknesses in policies and related institutions and praxis might arise; we then state three hypotheses to guide our strategy for data collection and analysis. Impediments to Better Outcomes Economic theory about market and government failures identifies several conceptual obstacles to a well-functioning WfD system. In their comprehensive review, Almeida et al. (2012), for example, discuss failures stemming from: (a) imperfections in capital markets which prevent individuals and firms from borrowing for training investments; (b) limited information, uncertainty, and myopia that distort the actions of individuals and firms with regard to training investments; (c) coordination failures arising from weak linkages among key stakeholders (for example, employers and training providers); and (d) labor market imperfections (for example, firms demanding suboptimal levels of training for their workers for fear of losing their trained workers to competitors). Theoretical considerations alone are insufficient, however, for designing policies to improve the system’s performance. Also needed is a deeper understanding of the specific policies and institutions that matter and the circumstances under which they matter. The evidence on this score is, unfortunately, sparse at present, particularly as it pertains to the experience of developing and emerging economies. We can nonetheless begin to contribute to the dialogue by creating and testing a new methodology that documents existing policies and institutions systematically and benchmarks them against practices in systems that have made significant progress over time. This approach is characteristic of the overall SABER initiative. Its application in multiple countries creates a comparative database that can be analyzed to narrow gaps in our knowledge about effective policies and institutions. Our data collection effort focuses on three areas where we believe market and government flaws are particularly relevant as impediments to better outcomes in workforce development: governance, 15 finance, and information. We briefly elaborate on them below to set the stage for discussing the design of our data collection instrument. Governance. Workforce development is a complex policy arena. It involves multiple stakeholders at different levels of decision-making whose goals, roles, and responsibilities may overlap or be in conflict (for example, ADB 2009 and 2010). At least four sets of stakeholders can be discerned: (a) public officials representing different ministries and levels of authority; (b) employers in their role (individually and through their trade associations) as a source of jobs, market intelligence, expertise, and advice to inform the training curricula, and possibly also of gifts and advocacy to support training programs; (c) training providers (again individually and corporately) who offer pre-employment, on-the-job and targeted training and career development services, networking among trainees, and a pool of potential recruits for employers; and (d) individuals who have dual roles, as trainees and as incumbent or future workers (who may be represented through unions), whether working for others or in self-employment. In this complex environment, appropriate governance arrangements can help clarify roles, responsibilities, and accountabilities among the stakeholders and enhance the quality of key relationships among them. Finance. All WfD systems depend on adequate and well-deployed resources, financial and non-financial, to function effectively. Funding arrangements—how money is mobilized, allocated, and channeled— shape incentives and therefore the decisions made by individuals, training providers, as well as employers and firms in their capacity as trainers and users of skills. Key design choices include: (a) the aggregate level and pattern of distribution in budget allocations; (b) the criteria for deciding on funding allocations, and on recipients’ access to and continuation of funding; and (c) diversity in the sources mobilized to support investments in WfD. Funding arrangements can exert an influence through channels on both the demand side and supply side and thus have the potential to help narrow gaps in skills supply and demand. In-kind resources, such as employers’ contribution of machinery or staff time for training purposes, are often difficult to document, but their presence is an important indicator of engagement of a key group of stakeholders. Information. Reliable and timely information provide signals that condition the behaviors of key actors on both the demand and supply sides in the market for skills. Useful information include the following: the skills gaps faced by employers; the menu and quality of training options and their costs; the employment and earnings of the graduates of training programs; the sources and availability of funds for training; and the trustworthiness of the formal certification of skills attained. The availability of such information can help influence the decisions of both individuals and training providers in aligning them to the demand for skills in the labor market. However, if an important economic goal of the country is to grow and upgrade into new and emerging areas of economic activity that require new technological, managerial, and operational capabilities, then passive information flows alone may not suffice to minimize skills gaps and mismatches. In such settings proactive coordination to bring the behaviors of the relevant parties into closer alignment with the areas of emerging skills demand may be appropriate. Hypotheses on Improving WfD Outcomes Improving the performance of the WfD system is akin to moving up the rungs of a ladder. For example, four rungs on the ladder corresponding to poor, modest, good, or great outcomes may be characterized as follows: • Poor outcomes: – Few employers value the WfD system; and – Interest in hiring graduates from the system is low. 16 • Modest outcomes: – System produces some useful outputs in terms of individuals trained; – Trainees and employers are generally dissatisfied; and – Access to training is limited and quality of skills imparted is poor. • Good outcomes: – System trains large numbers; – Most trainees and employers are satisfied with the system; – Some scope exists for continuous learning and skills upgrading; and – System lags behind in skills for innovation and technology upgrading. • Great outcomes: – System makes a difference and enjoys employers’ and trainees’ confidence; – Employers participate actively to provide feedback and support; – Graduates secure gainful employment; – System encourages continuous learning and skills upgrading; and – System adapts quickly to new economic conditions and opportunities. The challenge for policy makers is to situate their WfD system in the continuum of outcomes and to determine a context-appropriate strategy for progressing to the next rung of performance. Their task is arduous, given our current incomplete knowledge about effective institutions, policies, and praxis. An important objective of the SABER-WfD initiative is therefore to help narrow this knowledge gap by accumulating and analyzing the evidence in this regard. To guide our data collection effort and analytical strategy we start with three hypotheses on the links between policies and institutions in WfD and the system’s performance. Hypothesis 1: We contend that well-functioning WfD systems require mutually reinforcing policies that simultaneously address weaknesses in governance, finance, and information at multiple levels of decision-making. Policy gaps in any of these areas compromise outcomes in a consistent and similar way across countries. Accordingly, examining WfD policies in countries that have improved the performance of their WfD systems will provide insight into priorities for reform and suggest strategies for plugging the most damaging gaps at a given level of system development. Hypothesis 2: We suggest that good policies on their own are insufficient to improve WfD outcomes. Effective institutions for implementation and efficient feedback mechanisms are also critical in a dynamic reform agenda that delivers desired outcomes. These features combine to create a virtuous cycle of informed policy design, follow-up action, and self-correcting adjustments to minimize skills gaps and mismatches in a timely fashion. Suboptimal outcomes stem, on the other hand, from any or all of the following: weak or incomplete policy design, lack of attention to institutions and implementation, or disjointed feedback mechanisms. Hypothesis 3: We assert that a “learn-to-do and do-to-learn� approach is essential in the search for better outcomes in workforce development. It minimizes what some writers have referred to as the risk of “borrowing� policies that may have worked in one context and applying them in another with little adaptation or consideration of local conditions (for example, Allais 2010). The approach reinforces the notion that, in a field of still evolving knowledge, successful reforms will require intentional learning by national policy makers through systematic efforts to build domestic capacity for policy design and implementation. 17 The three hypotheses influence our approach in designing the SABER-WfD tool. The first explains the tool’s focus on the collection of comprehensive yet specific data on policies and institutions associated with various levels of decisions addressing market or government failures in governance, finance, and information. The second motivates the collection of evidence on institutions and policy implementation, the existence of policies on paper, as well as evidence on the scope of action. The third hypothesis leads us to document the use of feedback mechanisms and evidence of intentional learning in the process of policy reform and implementation. Through carefully crafted questions and answer options, the SABER-WfD instrument generates data for assessing a country’s WfD policies, institutions, and praxis against predefined rubrics (as explained in a later section and in Appendix 2). Suffice it to say that the rubrics describe a progression in scores corresponding to increasing evidence of institutionalized processes, of policy implementation informed by robust analysis, and of continuous learning through strong feedback mechanisms. Data for Benchmarking WfD Policies and Institutions The SABER-WfD tool uses a survey-like data collection instrument (DCI) to gather information on policies on workforce development, including formal training programs, non-formal programs and informal TVET, whether public or private, as well as employer-based training. 31 Because the respondent is not a person but a country (or subnational entities), an immediate question concerns the perspective from which the questions are meant to be answered. Keeping in mind the tool’s purpose of supporting policy dialogue, primarily among interlocutors in the human development sector, the most appropriate perspective is that of the line ministries responsible for WfD. These parties would typically be ministries of education and of labor, although other special agencies may also be relevant. Below we elaborate on the overall structure of the dataset generated by the DCI, explain its content, and summarize its key features. Overall Structure of the SABER-WfD Data Collection Instrument The DCI uses a cascading structure to organize the questions included in the tool. It first groups the questions into three broad functional dimensions. Each dimension is separated into policy goals that correspond to important thematic aspects of WfD institutions, policies and praxis. The questions under each policy goal are further grouped by topic, in order to create an efficient and natural flow in the line of inquiry during data collection. Details of key features of the DCI follow below. Three functional dimensions of WfD institutions, policies and praxis. The three dimensions are: (a) strategic framework, which sets the direction for WfD in relation to national goals for economic growth and productivity, and defines its authorizing environment; (b) system oversight, which relates to the governance of the WfD system and the arrangements that support its operational functions; and (c) service delivery, which pertains to the management of the provision of services, whether by public or 31 The use of a survey–like DCI adds to the toolkit for economic and sector work on WfD and facilitates cross-country exchanges; it also differentiates the SABER-WfD tool from other analytical approaches. For example, the European Training Foundation’s Torino Process (ETF 2012) shares SABER-WfD’s overall objective of informing policy dialogue. It offers a common analytical framework for country-specific analyses; and provides explicit guidance on the issues to be addressed and on the indicators for which data should be mobilized to prepare reports as a basis for dialogue. However, the Torino Process does not use a survey to collect data, nor does it rate countries’ WfD institutions and policy processes. 18 private provider, to achieve desired WfD outcomes on the ground (figure 2). 32 From the perspective of the line ministries, strategy is about sensing, influencing, and responding to the external environment for WfD; oversight is about defining the “rules of the game� for all stakeholders that have a direct interest and part in WfD activities; and delivery is about supervising the entities responsible for training provision. Figure 2: Functional Dimensions of WfD Policies and Institutions Strategy: Aligning WfD to national economic goals Delivery: Oversight: Managing for tangible Governing to achieve results on the ground desired goals Source: Authors’ construction. The three functional dimensions are interconnected, as the figure suggests. The quality of the strategic framework depends on the role that WfD is expected and enabled to play in fostering the economic success and well-being of individuals, businesses, and the country as a whole. The authorizing environment thus established sets the stage for defining specific institutions and mechanisms to guide the behaviors of the various parties involved—policy makers, employers, training providers, trainees. In turn, these oversight arrangements create the operational context for training provision by public as well as private providers. These service providers’ success in equipping trainees with skills for employment and productivity provides the feedback that closes the policy-making loop. In essence, the DCI collects data that describe the WfD institutions and policies through which governments act to enhance the workings of two markets: “the [labor] market in which employers and individuals trade work for wages [or earnings]; and the training market in which individuals obtain training [and therefore acquire skills] from training providers� (Karmel 2011). The instrument includes questions on the basis of their likely relevance in most developing and emerging economies. 32 The functional dimensions of policies in figure 2 bears some resemblance to the threefold challenge that the UKCES (2009) believes the United Kingdom faces in its attempt to create a better, more responsive, demand-led, and effective WfD system. The three challenges are: (a) “a ‘policy gap’, where a stronger emphasis on the ‘demand’ side is required to bring about better balance to the policy agenda�; (b) “a ‘measurement gap’, where developing a more appropriate suite of ‘success measures’ for the system would help better align policy with the 2020 Ambition, and policy with delivery�; and (c) “a ‘policy to practice gap’, where delivery and arrangements on the ground do not always fulfill the ambition of the policy promise.� 19 Drilling down to the policy goals. The DCI’s questions about WfD institutions, policies and praxis are grouped into nine policy goals (figure 3), three each for the three functional dimensions. While other ways exist to organize the DCI, the chosen structure satisfies the criteria of simplicity and natural sequencing of the questions. It is useful to note briefly here some improvements to the instrument based on feedback from the pilot phase. DCI Version 2.5 excludes or consolidates questions found to be duplicative, and it rewords questions for greater precision where this is required. The answer options have also been adjusted in this spirit. The added precision and parsimony led to a slight reordering of the policy goals under the functional dimensions relating to oversight and delivery (see note to figure 3 below for details). It also prompted a reduction in the number of policy actions to 18 from the 27 in the pilot version of the tool. Appendix 1 provides a succinct summary of the organization of these 18 actions and the associated 47 topics for which data are collected. Below we explain the motivation for the specific questions posed in the DCI. Figure 3: Policy Goals under Each Dimension Strategy Oversight Delivery Direction Resources Diversity Demand-led Standards Relevance Coordination Pathways Results Source: Authors’ construction. Note: The single-word labels for the nine policy goals have specific meanings that are explained in the text below. The policy goals for two of the three function dimensions (viz., oversight and delivery) are arranged slightly differently here than in version 1.2 of the DCI used in the pilot phase. For oversight, the previous version ordered the policy goals as follows: pathways, resources and standards; for delivery, the previous order was: relevance, diversity and results. Readers are advised to keep these differences in mind in reading the reports from the pilot phase of the project. Dimension 1: Strategic Framework WfD is not an end in itself but an input toward broader goals—of boosting employability and productivity; of relieving skills constraints on business growth and development; and of advancing overall economic growth and social wellbeing. Its multidimensional nature poses complex challenges and tradeoffs. For example, when policy makers refer to skills for growth, they are typically concerned about skills that can help the economy diversify, innovate, and upgrade its technological capability. 20 However, because these skills pertain, by and large, to formal sector jobs they may be irrelevant to the majority of workers in developing countries who earn their living in the informal sector. A second source of complexity is that all jobs call for a combination of skills. Bloom’s (1976) classic taxonomy makes a distinction among cognitive, affective, and psychomotor skills. 33 Development of these skills typically involves different periods of gestation; the sequencing and timing of investments also depends on the specific skills required in different jobs. Some skills (for example, basic literacy and numeracy) entail sustained effort from relatively young ages (for example, through early childhood education and general basic education); others call for specific programs that meet employers’ more immediate needs. On a fundamental level, WfD strategies entail trade-offs between short- and longer- term objectives, often because resources are limited and because of differences in the pattern of costs and benefits across investments in skills development. A third reason for the complexity of WfD is that outcomes depend on the actions of multiple stakeholders, including individuals, employers, training providers, and government officials (often from several ministries or agencies). Asymmetric access to credit, imperfect or incomplete information, misalignment of incentives and weak cooperation among these actors are some of the obstacles that can lead to suboptimal outcomes in WfD. 34 These complex objectives and challenges of WfD lead us to emphasize the following policy goals under the functional dimension relating to strategy: (a) the setting of a strategic direction for WfD in alignment to the country’s goals for economic growth, poverty reduction and social development, (b) prioritization of a demand-led approach to WfD, and (c) facilitation of critical coordination among the actors at the leadership level of decision-making. Bringing together these elements requires visible leadership and advocacy based on an understanding of and conviction about the role and nature of WfD in national development. It necessitates sound judgment in identifying strategic priorities and in mobilizing collaborative effort to solve critical short-run challenges while keeping an eye on longer-run ambitions for WfD and national socioeconomic goals. This calls for effort to remove practical roadblocks to cross- sector cooperation among diverse stakeholders where such cooperation is vital for achieving critical breakthroughs. The DCI builds on these ideas and includes questions that are grouped under three policy goals, as elaborated below. Policy Goal 1: Setting a strategic direction for WfD. In reviewing the pattern of economic growth over the past few decades, the Commission for Growth and Development (2008) has emphasized the importance of “leaders who are committed to achieving growth and who can take advantage of opportunities from the global economy.� 35 In the area of workforce development to support growth, 33 Cognitive skills refer to a person’s grasp of academic disciplines such as English, mathematics, as well as various pure and applied sciences, history, and so forth. Affective refer to his or her perceptions about work, concept of self and others, and attitudes toward timeliness, accuracy, quality, and performance. Psychomotor skills refer to the possession of skills required to perform the tasks or duties involved in an occupation, job, or business (for example, operating a lathe, preparing architectural plans, installing equipment, and so forth). The Dictionary of Occupational Titles (DOT) in the United States makes a distinction among jobs according to the degree of involvement with Data, People, and Things (see http://www.oalj.dol.gov/PUBLIC/DOT/REFERENCES/DOTAPPB.HTM (accessed on October 10, 2011). The O*NET’s Content Model, which replaced the DOT in 1998, continues to emphasize the multifaceted skills requirements of jobs (for more information, see http://www.onetcenter.org/content.html (accessed on October 10, 2011). 34 See Almeida et al. (2012) for an elaboration of various market and government failures that can lead to suboptimal outcomes in workforce development. 35 See http://www.growthcommission.org/index.php?Itemid=169&id=96&option=com_content&task=view (accessed on November 9, 2011). 21 East Asia’s fastest growing economies over the past 50 years— South Korea; Taiwan, China; and Singapore among others—provide the clearest examples of the role of leaders in shaping the skills agenda (Commission on Growth and Development 2010). These nations have built a strong foundation in basic skills through general education for all and now consistently appear among the top-performing nations in international assessments (OECD 2010b; Mourshed et al. 2010). At the same time, they have also taken decisions to develop training systems that effectively equip workers with job-relevant skills to expand their own technological capabilities (see, for example, Abe 2006; Ashton et al. 2002; Green et al. 1999; Kuruvilla et al. 2002). The increasing sophistication of the exports from these countries is one sign of success. In South Korea, for example, electronic, electrical, and other high-technology products made up nearly 40 percent of the country’s exports in 2006, up from less than 5 percent in 1980 (Yusuf and Nabeshima 2010). Bearing in mind the important role of leaders, the DCI’s questions for this policy goal seek information on the advocacy for WfD by top-level national leaders, whether in government or the private sector. 36 Such advocacy turns the public’s attention onto WfD and helps to legitimize WfD as a priority for economic development. It also ensures that the subject receives due consideration in policy dialogue and in the implementation of agreed policies. The focus of top-level leaders may relate to broad challenges in WfD that are long term in nature (e.g., investing in early childhood development) or to more specific and immediate concerns with technical and vocational skills (e.g., skills required to support moving up the value chain in a few strategic economic sectors). The commitment of top-level leaders to any particular cause is admittedly difficult to quantify. For WfD, we document it through questions that reveal: (a) the extent of collaboration on shared strategic priorities among the champions of WfD; (b) the specificity of their action to advance the agenda for WfD; and (c) the attention to follow- up through systematic monitoring and tracking of implementation progress. Policy Goal 2: Prioritizing a demand-led approach to WfD. To be effective, advocacy for WfD must be based on credible assessments of the demand for skills in light of a country’s economic prospects (ILO 2008). It also recognizes that employers can provide an important source of reality check on such assessments. The specific ways by which the government brings these elements together in prioritizing a demand-led approach to WfD depends on the country’s circumstances. To illustrate, in 1996 Costa Rica’s President led the country on an eventually successful bid to persuade Intel, a global computer chip manufacturer, to set up the firm’s newest plant in the country (Spar 1998; World Bank/MIGA 2006). Part of the deal included the introduction of new courses in technical training and new curricula. 37 In subsequent years, the country continued to align its education and training programs to meet the needs of employers in the information technology industry. 38 The IT industry is today a major part the 36 The previous version of the DCI also included under this policy goal questions on evaluation of the economic prospects for skills and on policies to align skills demand and supply. These questions are now consolidated in DCI Version 2.5 with those for the next policy goal, in order to streamline the instrument. 37 The Technological Institute of Costa Rica (ITCR) introduced a one-year certificate (for high school graduates), and a one- year associate degree and English language training programs. The technical courses focused on new fields such as semiconductor manufacturing and microelectronics, and later also included materials science (Spar 1998; World Bank/MIGA 2006). 38 These demand-responsive efforts included, for example, the following: (a) new technical programs and enhanced curricula at the country’s leading educational institutions— ITCR, the University of Costa Rica (UCR), and the National Training Institute (INA), especially during 1999–2003; (b) an English reinforcement program at ITCR; (c) links between UCR’s School of Physics and technical and vocational schools for electronics; and (d) outreach programs in technology and science for teachers and students in primary, middle, and high schools. 22 economy, accounting for nearly 19 percent of exports in 2009, compared with just 4 percent in 1997. 39 Mexico’s Integral Quality and Modernization (CIMO) program is another example of government effort to foster skills upgrading among firms. 40 In South Korea, the introduction in 2010 of a high-profile government initiative involving “meister� high schools exemplifies an effort to address emerging shortages of technicians for the country’s priority sectors (for example, electricity, media, logistics, shipbuilding, IT, and automotive). 41 Two aspects of a demand-led approach to WfD receive emphasis in the questions included in the DCI. The first pertains to the clarity on the demand for skills and areas of critical constraint. The questions seek evidence on the existence of and basis for formal assessments of the country’s economic prospects and their implications for skills. They also document more specific skills constraints in the country’s priority economic sectors. The second aspect of a demand-led approach to WfD focuses on the extent to which employers, including those in the informal sector, are involved in shaping the country’s strategic priorities for WfD and in supporting skills-upgrading for workers. Besides seeking confirmation of employers’ involvement, the DCI’s questions also gather information on the incentives available to employers to upgrade the skills of workers. Such information sheds light on the extent to which the ambition of skills-led economic development is in fact backed up by policies to intensify employers’ and firms’ demands for more and better skills to help raise productivity. Policy Goal 3: Strengthening critical coordination. WfD involves a wide range of activities and an extensive web of stakeholders with diverse interests, roles, and responsibilities. Ensuring that the combined effort of these partners is consistent with the country’s key priorities for WfD is therefore an important goal of strategic coordination. Such coordination typically requires leadership at a sufficiently high level to overcome barriers to cross-sector or cross-ministerial cooperation that is often impossible to resolve among peers with similar levels of decision-making authority. Fundamentally, the core issues pertain to the “interactions among institutions, processes and traditions that determine how power is exercised, how decisions are taken on issues of public and often private concern, and how citizens and other stakeholders have their say� (Abrams et al. 2003). An arrangement that clarifies roles and responsibilities and fosters communication among the relevant WfD stakeholders sets the stage for coherent and well-coordinated actions by these players (see, for example, ADB 2009). Coordinated effort is particularly relevant for major initiatives in WfD that are breaking new ground, that involve new or nontraditional partners, and that possibly also introduces new operational procedures. Such coordination at both the strategic and operational levels can help avoid duplication—or worse, misalignment or conflict—of effort and ensure that the most critical initiatives receive the necessary moral and material support for experimentation, consolidation, and maturation. Ireland’s experience with the establishment of new regional technical colleges (RTCs) beginning in the late 1960s is an example of coordinated effort to supply mid-level technicians for the country’s emerging industries 39 Data from World Development Indicators (see http://data.worldbank.org/data-catalog/world-development- indicators/wdi-2011, accessed November 27, 2011). 40 For more information about CIMO, see Tan et al. 2004. 41 These shortages stem from the preference of young Koreans and their families for studies at the universities. The “meister� high schools were created to offer students an attractive non-university route to the labor market. These schools offer new curricula that provide high quality technical training and explicitly cultivate “21st century skills.� A personal visit in May 2011 by three of this paper’s authors to Sudo Electric High School in Seoul, one of the first “meister� schools that the country’s President inaugurated in March 2010, found the following: (a) a student-centered pedagogy; (b) inclusion of training for personality development; (c) opportunities for students to study abroad; (d) English used as the medium of instruction by teachers who are competent in the language; and (e) strong linkages with specific employers. 23 (O’Hare 2008). 42 The Indian government’s decision to create several new bodies in a three-tiered governance arrangement for WfD—the National Council on Skill Development, chaired by the Prime Minister; the National Skill Development Coordination Board, chaired by the deputy chairman of the Planning Commission; and the National Skill Development Corporation, headed by an eminent private sector industrialist—exemplifies a more recent effort of strategic coordination at the highest level of government. 43 To document the status of coordination among key parties involved in WfD, the DCI’s questions probe for information on the roles played by government and non-government stakeholders and on the mechanisms for coordination among them. They also seek evidence of coordinated effort among the advocates of WfD on major strategic priorities and of explicit mechanisms for tracking implementation progress. Dimension 2: System Oversight The second functional dimension of WfD systems addressed by the SABER-WfD tool concerns the arrangements for oversight. These arrangements define the “rules of the game� for key stakeholders in WfD, including individuals, training providers and employers. Although WfD systems cover, by definition, all levels and types of training provision, the SABER-WfD tool focuses on oversight of those parts of the system that equip individuals with the vocational and technical skills necessary for semi- skilled and skilled jobs. Typically, training is offered at the secondary and post-secondary levels; in more mature and sophisticated economies, it is also offered at the tertiary levels, including in technical universities. These levels may be included, as appropriate, when using the SABER-WfD tool to document the system’s oversight arrangements. The tool recognizes that skills acquisition occurs through a variety of channels and modalities, including: (a) pre-employment or initial technical and vocational education and training; (b) continuing training and skills upgrading for those already in the workforce, through formal apprenticeships and other on-the-job training, formal course work, or nonformal or informal methods; and (c) training components in active labor market programs designed for target populations. The main objectives of oversight are to facilitate efficient and effective skills acquisition by individuals to improve their employability and productivity, and to enable employers to meet their demand for skilled workers in a timely manner. The goal is to minimize mismatches in skills supply and demand, thereby enhancing the contribution of WfD to economic growth and social progress. In countries with large informal sectors where most people are self-employed, the oversight function also entails ensuring that individuals are able to access training services that are effective in enhancing their earning power. If the markets for skills functioned perfectly there would be a limited role for oversight. Individuals could easily mobilize the funds required to pay for their training; they would have access at no cost to accurate and timely information about the types of skills in demand; they would know where and how to get trained, including the pre-requisites for enrolling in a desired program; and once trained, they would be able to send accurate and credible signals to potential employers regarding the skills they 42 O’Hare (2008) elaborates that the Ireland Development Authority (renamed since 1994 as the Industrial Development Agency (Ireland)), in its effort to convince foreign businessmen to invest in the country, used a portfolio of incentives to encourage a greater output of skilled technicians throughout Ireland, while simultaneously involving the leaders and staff of the training institutions to participate actively in its promotional efforts (for example, by attending functions where foreign businessmen were gathered, and travelling to events in potential investor countries). 43 For more details of the arrangements in India, see Planning Commission, Government of India (2012); http://labour.nic.in/upload/uploadfiles/files/Policies/NationalSkillDevelopmentPolicyMar09.pdf (accessed January 7, 2013). 24 possess. Likewise, employers would have reliable and timely information about the skills of prospective and incumbent employees; they would be able to make accurate choices among possible training options for their workers; and they would face no constraints in organizing or purchasing training. Training providers, for their part, would be fully aware of the skills that are in demand and would tailor their programs accordingly; and they would have the information to create efficient pathways for skills acquisition for their clients, for example, through articulation agreements with providers that offer complementary services. This scenario is unrealistic and highly unlikely to emerge on its own. Financial constraints are an obvious barrier, particularly for disadvantaged individuals or small firms, which are likely to prevent potential beneficiaries of training from making the investment, thus leading to suboptimal investments in WfD (Almeida et al 2012; Finegold 1995). Information is also costly to generate and acquire, and it may not be complete, relevant, accurate, or timely (see, for example, Woods and O’Leary 2007). Individual firms often cannot spare the time to codify their skills requirements, particularly if they employ few people or operate in a fast-changing business climate. Other distortions may arise from individuals succumbing to the temptation to overstate their knowledge and skills in order to land a desirable job or a promotion. Likewise, training providers may be motivated to mask the quality of their program offerings in the interest of attracting and retaining their clientele. In light of the foregoing ideas, the SABER-WfD tool’s questions on the oversight dimension of its analytical framework are organized around the following three policy goals: (a) ensuring efficiency and equity in funding WfD investments, which fosters the system’s long-term financial sustainability; (b) assuring relevant and reliable standards for quality in service provision and skills acquisition, which creates more efficient signals in the market for skills; and (c) diversifying pathways for skills acquisition, which encourages lifelong learning as a basis for adapting to evolving labor market conditions. Policy Goal 4: Ensuring efficiency and equity in funding for WfD. In practice, funding is a ubiquitous concern in WfD systems (see, for example, Wolf and Erdle 2009). In poorly funded systems, providers are hard pressed to satisfy the expectations of employers who in turn distance themselves from the providers; the result is to reinforce providers’ tendency toward supply-driven program offerings, which further undermines their ability to respond to employers’ needs. These conditions reduce the potential for productive partnerships in WfD. Many systems in developing and emerging economies fall victim to this vicious circle because governments simply do not have enough resources to prioritize TVET when other parts of the education system, such as primary and secondary schooling, are also poorly funded (see, for example, Atchoarena 2002; Dunbar 2011). Support for TVET is moreover often weakened by the fact that most students come from poor families that typically lack the political clout to influence budget allocations, as well as by the general perception of TVET as a “second-class� route to the labor market with doubtful impact in equipping students with job-relevant skills. Some countries have nonetheless successfully tackled some of these problems. In Switzerland, for example, TVET tracks at the secondary level offer excellent programs and flexible pathways to tertiary level courses, which attract a sufficiently meaningful share of the country’s top students to overcome public bias against such programs (OECD 2010a; FOPET 2012). In Singapore, sizable investments in a high-quality TVET system over the years, coupled with sustained attention to the employability of graduates, have lowered social resistance to the TVET programs that the less academically inclined students enter after 10 years of general schooling (Law 2008). 25 With regard to funding for the training of incumbent workers, a common arrangement involves collecting a tax or levy from firms, often in relation to the size of the payroll, and using the proceeds to reimburse firms that train (Johanson 2009). Some observers cite Malaysia and South Korea as examples of successful schemes (for example, Lee 2009; Tan 2001). As for allocating funding to achieve results, having transparent and explicit allocation criteria and ensuring that the criteria themselves are reviewed and adjusted in light of experience, are a sine qua non of good oversight of WfD resources. Accordingly, some recent World Bank operations that support TVET, for example in Pakistan and India, seek to foster these practices.44 Regarding partnerships, the relationship between Costa Rica’s training institutions and Intel mentioned earlier exemplifies arrangements that have been sustained over time because they generate benefits for all parties: the firm, the participating training institutions, and the trainees (e.g., Linburg 2009). The underlying policy design challenge is to influence WfD catalytically through the government’s role in resource mobilization and funding WfD, ensuring efficient and effective use of the available funds, and in fostering partnerships that can multiply the resources, pecuniary and otherwise, that are available to encourage and support investment in WfD by individuals and employers. Governments fund WfD in part to counteract possible under-investment because of credit constraints (for example, as experienced by individuals from poor families), myopia on the part of prospective trainees and firms, incomplete information, or other imperfections in the market for training. Because funding for WfD is required on a continuing basis while public budgets are often limited and subject to intense competition from other competing projects, a clear strategy for sustaining support for WfD, one that benefits from continuous assessment and adjustment, can help ensure that funding is stable, sustainable, and sufficient to achieve the most important goals at reasonable cost. Such a strategy would recognize that besides public budget allocations, resources for WfD may also come from fees paid by trainees or their sponsors, and contributions by private entities, nongovernmental organizations, local communities, and/or external donors. Some of the contribution may take the form of in-kind donations of inputs that are essential for effective training programs (e.g., equipment and the services of experienced skilled workers). To keep the DCI parsimonious, its questions for the policy goal of ensuring efficient and equitable funding for WfD seek to shed light mainly on: (a) the overall landscape of financing for initial and continuing technical and vocational education as well as the financing of targeted services for disadvantaged groups; (a) the prevalence of criteria-based allocations; (c) the practice of routine monitoring of the impact of funding arrangements on efficiency and equity; and (d) the nature and effectiveness of partnerships with employers to enhance the system’s financial sustainability. Policy Goal 5: Assuring relevant and reliable standards for quality in WfD. The underlying issue is trust in the skills acquired by individuals and relatedly, in the services offered by training providers. Allais (2010) notes that at present, more than 100 countries are in the process of establishing a national qualifications framework. 45 Her review of experiences in 16 of these countries cautions against overly 44 For example, the Sindh Skills Development Project and the India Vocational Training Improvement Project; more information can be found on these projects at, respectively, the following sites: http://web.worldbank.org/external/projects/main?pagePK=64283627&piPK=73230&theSitePK=40941&menuPK=228424&Proj ectid=P118177; and http://web.worldbank.org/external/projects/main?pagePK=64283627&piPK=73230&theSitePK=40941&menuPK=228424&Proj ectid=P099047 (accessed on October 13, 2011). 45 Part of the interest stems from employers in countries that employ foreign workers. For these employers, the qualifications framework in the sending countries is a tool for gauging the quality of potential recruits from these countries. Another 26 ambitious expectations in the short run. Success requires significant resources and patient work by skilled personnel over many years—conditions that are often absent in developing and emerging economies. Ganzglass et al. (2011) stress the importance of stakeholder involvement in setting standards, of benchmarking standards to employers’ requirements, and of reliable processes for validating standards and assessing learning. The system adopted by the National Institute for Metalworking Skills in the United States is an example of the application of these principles. 46 In less developed nations, where the informal sector dominates the economy, the testing and certifying of skills present special challenges, in part because many skills are acquired through traditional apprenticeships and standards are often tacit and probably uneven across practitioners (Atchoarena 2002; de Largentaye 2009; Janjua and Naveed 2009). Nonetheless, in the Indian government’s nationwide Modular Employable Scheme, launched in 2010, a key feature is to codify the standards for skills acquisition and certification for a wide spectrum of jobs, many of them in the informal sector. 47 Like qualifications frameworks, accreditation is also about standards, but its focus is on training providers or programs. The process of accreditation typically involves an authorized accrediting body certifying, usually on a term-limited and renewable basis, that the training provider has met externally established standards for excellence or quality in service delivery (Vlăsceanu et al. 2007). The relevance of accreditation has grown in recent years against a backdrop of rapid expansion of enrollments at all levels of education and training in developing countries. The trend has overburdened public institutions and stimulated an explosive growth of nonstate providers (see, for example, King and Palmer 2010). The diversity in options for WfD investments is a positive development, but in the absence of adequate standards or regulation it could also mean a proliferation of substandard providers and programs (see, for example, Atchoarena 2002; Dunbar 2011). Specifying minimum standards is one way to weed out the worst operators while allowing others to reach for a higher bar, if desired—for example, by obtaining accreditation with internationally recognized bodies such as the International Standards Organization (ISO). 48 To document the institutions and policies that a country has put in place to advance the goal of assuring relevant and reliable standards, the SABER-WfD DCI contains questions covering three broad areas of interest: (a) the scope and robustness of qualifications frameworks; (b) the protocols for and credibility of skills testing and certification; and (c) the specification and enforcement of accreditation standards and the support and incentives available for training providers to meet the desired standards. Policy Goal 6: Diversifying the pathways for skills acquisition. Technological advances affect how economic activity and work is organized in an increasingly integrated global economy. Among other impacts, they are likely to make job markets more volatile, notably by destroying jobs in sectors that are no longer competitive and creating new ones elsewhere in the economy, all within shorter and shorter time intervals. Because the new jobs are likely to require new skills and competencies, workers will need to keep their skills up-to-date and attuned to fast-changing economic conditions in order to remain employed. In aging societies, where retirement may be delayed to keep pension schemes financially solvent, these trends mean that more and more people can no longer count on working in a single job or even in a single career over the course of their (extended) working lives (see, for example, Kiley and source of interest stems from the growing demand for intercollegiate credit transfers and articulations across programs and institutions. 46 National Institute for Metalworking Skills. Undated. “Skills Standards.� https://www.nims-skills.org/web/nims/7 (accessed June 17, 2011). 47 See http://dget.nic.in/mes/sectors.htm (accessed on July 30. 2011). 48 Singh and Sareen (2006) note the rising popularity of ISO 9000 certification among Indian educational institutions. 27 Cannon 2000; Wang 2012). For growing numbers of people, therefore, life-long learning is not merely an attractive concept but an economic necessity. Lifelong learning receives the greatest support when clear pathways exist for skills acquisition and when students are able to transfer across courses, progress to higher levels of training, or gain access to programs in other fields (MacKenzie and Plovere 2009). In reality various barriers restrict the available pathways. An important barrier is the “dead-end� status of some courses of pre-employment education and training. 49 In many countries, students who pursue technical and vocational courses of study often earn credentials that receive little or no recognition for admission into courses of further studies. Often the problem is not just the poor image of these courses but also their intrinsic poor quality. Switzerland is an example of a country with an effective arrangement to address this problem. Its system of technical and vocational secondary schools is sufficiently attractive that they enroll some of the best students. Further, the baccalaureate certificate that students earn at the end of their studies is valid for admission to the country’s various universities (FOPET 2012). In the United States, the arrangements are less formal, but many community colleges offer flexible pathways through such practices as: (a) giving students credit for work experience in industry; (b) collaborating with high schools on preparatory courses that allow students to fulfill the prerequisites for full-time enrollment in a community college; and (c) entering into agreements with universities and other colleges, most often within their respective states, to provide students with the option to continue on to four-year college- or university-level programs once they have proven themselves in a community college program. 50 A second barrier to lifelong learning, especially for those already in the workforce, is the absence of arrangements to recognize prior learning. Individuals without the formal credentials for admission into their chosen training programs are assumed to have no relevant skills and must thus start their training at a more basic level than necessary. The result is to prolong training time and increase costs for prospective trainees, which may in turn discourage them from embarking on plans to learn new skills or build on existing ones. Thus, even for the motivated, lifelong learning may be haphazard or even impractical under these circumstances. Ireland is an example of a country where recognition of prior learning has been institutionalized through a well-structured national qualifications framework. 51 The public is informed about the relations among qualifications and certificates earned at various levels of instruction, from the pre-secondary to the university; and individuals can obtain help with qualification recognition from the agency in charge.52 Disadvantaged population groups face bigger challenges in pursuing lifelong learning and may therefore require targeted services, such as special outreach programs or customized information. Reflecting the foregoing considerations, the DCI’s questions for the policy goal of diversifying pathways for skills acquisition seek information that would permit an assessment of the following three broad aspects of the system: (a) the extent that it promotes educational progression through multiple learning pathways, including for students in TVET streams; (b) the facilitation of lifelong learning through the 49 For discussion of this issue in developed and developing country contexts, see, for example Mupinga et al 2006, Sliwka 2010, and Wolff 2011. 50 For example, the community colleges in the states of Virginia and Maryland have such agreements with their respective state colleges and universities. 51 Its system fits what is described by Hanf and Hippach-Schneider (2005): the framework essentially “takes all of … the formally recognized qualifications and arranges them in a clearly defined structure…. [Qualifications are] understood as sets of certified or documented skills, with no regard given to the respective learning path.� 52 See http://www.qqi.ie and http://www.qualificationsrecognition.ie/, accessed December 21, 2012. 28 recognition of prior learning and related mechanisms; and (b) the availability of support services to encourage lifelong learning, particularly among the disadvantaged. Dimension 3: Service Delivery This functional dimension of the WfD system shifts our focus from the strategic and systemic to the operational. It addresses implementation challenges in WfD that stand between a country’s ambition for WfD to support economic development and what materializes in reality. Training providers, both government and nonstate, are the main channels through which the country’s strategic and systemic policies are translated into operational applications. Thus, we disaggregate this policy dimension into the following three policy goals: (a) promoting diversity and excellence in training; (b) fostering relevance in public training programs; and (c) enhancing accountability for results in WfD throughout the system. Policy Goal 7: Enabling diversity and excellence in training provision. Because the demand for skills is impossible to predict with a high degree of precision, a top-down approach to training provision is often ineffective. Instead, having a diversity of providers that are appropriately motivated to meet the demand for skills is arguably more promising (see, for example, Booth and Snower 1996). Most systems indeed include both state and nonstate providers, thus offering individuals and firms a range of choices in the training market. Encouraging excellence in training provision requires different interventions, however, for state and nonstate providers. Among nonstate providers, financial viability is a powerful incentive that conditions their behavior. Providers that depend largely on fees paid by individuals or firms (typically on behalf of their employees) have strong incentives to explore the specific skills required by employers and tailor their training programs accordingly. By working closely with employers and developing a reputation for responsive and effective training, these providers can improve the attractiveness of their services, thereby assuring the commercial viability of their business. A downside of the profit motive, however, is that providers may cut corners to save costs or hide serious flaws in their operations that are difficult for a poorly informed public to detect. In some countries, weak regulation of standards in a context of rapid expansion has led to a proliferation of private sector providers, many of them offering substandard services (see, for example, Atchoarena 2002; Dunbar 2011). User-driven models of financing are one way to promote diversity in training provision while muting the potential adverse effects of the profit motive on quality. In Australia, the state of Victoria introduced a model that the government argues would “drive greater competition (thereby achieving increased accountability, accessibility and quality) and respond more effectively to changing labour market needs� (Skills Victoria 2011). In Kenya, a large-scale voucher program for workers in the informal sector (also known as the “Jua Kali� sector) led providers to tailor their programs to the unique needs of informal sector workers (Johanson and Adams 2004). More recently, in 2008, another experiment with vouchers in Kenya suggests that institutions enrolling voucher winners were more likely to expand their course offerings compared to those that enrolled no voucher recipients (Hicks et al. 2011). In Chile, only providers registered with the National Training Authority (Servicio Nacional de Capacitacion y Empleo– SENCE) are eligible to bid for publicly-funded training programs (Johanson 2009). Among publicly-funded providers of training, a key aspect of quality is their responsiveness to the demand for skills from employers and students (ADB 2009; Dunbar 2011). In many developing and emerging economies, weakness in this area stems in part from chronic shortages of public funding, a constraint that limits the range and quality of programs that providers are in fact able to offer. Rigidity 29 in other areas, such as bureaucratic decision-making, compounds the challenges. The result is often a vicious circle of overly centralized controls which stifle initiative and institutional flexibility and weaken employer interest in the training programs. The loss of employer engagement means that training providers will, at best, be slow in addressing employers’ demands for skills and, at worst, be uninformed about competencies desired by employers. The governance of publicly-funded providers is at the core of the issue here. Granting autonomy to publicly-funded institutions is one approach taken by various countries to improve the performance of public institutions. In India, under an ongoing modernization program of the country’s more than 1,800 industrial training institutes, institute-level management committees, each chaired by a prominent local industry leader and comprising government and industry members, are being set up and granted academic and financial autonomy to operate the institutions (Prasad 2012). In the United States, most of California’s community colleges have institutionalized advisory boards with industry representatives as members. The colleges rely on their respective boards for guidance and inputs to ensure that course offerings are relevant to industry needs (Lindburg (2009). In Singapore, the Institute of Technical Education is constituted as a statutory board under the Ministry of Education; the Institute enjoys a high degree of management autonomy (e.g., over staffing and course offerings) and is accountable for achieving targets agreed with its parent ministry (Law 2013 forthcoming). In light of the ideas discussed above, the SABER-WfD tool’s questions for assessing the goal of promoting diversity and excellence across training providers cover the following: (a) the role of nonstate training provision, arrangements for quality assurance and evidence of continuing reviews of policies toward nonstate training providers; and (b) the autonomy and accountability of public training providers. Policy Goal 8: Fostering relevance in public training programs. Making this an explicit policy goal recognizes the fact that the government in many countries directly operates publicly-funded training institutions. The scope of government involvement varies widely: in Chile, for example, government-run training institutions exist only at the secondary level while in Costa Rica such institutions offer training at both the secondary and post-secondary levels. The market relevance of public training programs depends on providers’ access to reliable sources of up- to-date information on current and emerging skills demands. Employers and industry associations are a key source of such information. Without systematic arrangements, however, it may be difficult and probably too costly for individual training providers to gather the desired information on a continuing basis. In the Netherlands, the problem is solved through an institutionalized arrangement that involves an umbrella organization connecting training providers to specific employer-led sector skills councils that define occupational standards and competencies to shape the curriculum used by the training providers (Ashton 2006). Another source of information, particularly in relation to future skills needs, is research institutions. Such institutions are at the forefront of developing, adapting, and introducing products and services new to the country or region; their activities can therefore offer useful insights into the future demand for skills. In Australia, the training of technicians for the wine industry benefits from a close connection between the providers serving this industry and the relevant research institutions (Ferrier et al 2003; Anderson 2010). In China, the Beijing Agricultural School (BAS) disseminates the fruits of agro- technology research in the agricultural sector and upgrades the technological competencies of farmers through formal and nonformal training programs (UNESCO 2002). 30 Getting the curriculum right is only half the battle. Also needed to deliver market-relevant public training programs are instructors and administrators with relevant exposure, if not actual work experience, as well as well as access to facilities, equipment, and instructional materials. Because the costs of hiring staff with industry experience are high, possibly unaffordable for most institutions, it is important to pursue a variety of arrangements to make training as relevant to industry needs as possible. In-service training to help administrators and instructors keep their knowledge of industry trends up to date is an important part of the solution (ILO 2011; Toze et al 2009). The inclusion of workplace training is also important. In some systems, publicly-funded training colleges deliver theoretical off-the-job training while employers provide the on-the-job component, through such arrangements as apprenticeships, internships, or other work place opportunities for learners to gain authentic experience in performing jobs or in handling production equipment. Germany’s dual training system is perhaps the best known example of such arrangements (e.g., Bosch 2012). Workplace training also features in other well-developed WfD systems (e.g., the Netherlands and Australia) although the specific arrangement may differ from Germany’s. The relationship between the Korea University of Technology and Education (KUT) and Samsung illustrates an arrangement where various elements work together to enhance the relevance of training programs in a public institution. 53 KUT hosts the Employee Vocational Education Program, which offers short courses to Samsung’s own workers as well as those from the firm’s subcontractors. KUT provides the premises, and the teaching and administrative staff. Samsung, for its part, leads in defining the course curricula, provides the equipment, and lends in-house experts to co-teach the courses. The collaboration benefits all parties. Samsung gains from having subcontractors that can meet its product specification and quality standards; its subcontractors gain from being able to retain or expand their business with the company. Most importantly, the university gains from the opportunity to involve its faculty in organizing and teaching the courses. This involvement enhances the competence of its administrators and instructors in the design and delivery of industry-relevant training programs. The foregoing examples motivate the inclusion of questions in the SABER-WfD tool designed to gather information on: (a) the existence and nature of the links between public training institutions and industry partners, and between training and research institutions; (b) industry’s role in defining training curricula and other related matters; and (c) the criteria for staff recruitment and the availability and practices regarding in-service staff development. Policy Goal 9: Enhancing evidence-based accountability for results. Systematic monitoring and evaluation of service delivery are hallmarks of a strong WfD system. The types of data collected and how they are used are indicative of the commitment to ensuring that training providers and the WfD system as a whole are delivering on expected outcomes. Requiring training providers to report administrative information about their operations fills a basic and useful purpose for the government authority mandated to oversee service provision. Even simple reporting on enrollments, staffing, budgets and main accomplishments, if done routinely, serves as a tool for accountability. In Singapore, public sector training providers typically issue annual reports in this spirit. 54 If all providers, whether state or nonstate, report on their operations, the information, 53 This information is based on a personal visit to KUT in May 2011 by three of the authors of this paper. 54 For examples, see the annual reports of the Singapore Workforce Development Authority at http://www.wda.gov.sg/content/wdawebsite/L209-001About-Us/L209D-AnnualReports.html (accessed August 12, 2012). 31 when consolidated into an accessible database, can clarify key features of and trends in training provision, including differences in performance across training providers. South Korea is an example of a country where policy makers have access to a compilation of comprehensive and up-to-date data from administrative and other sources to inform policy design (see, for example, KRIVET 2007). Beyond information on training provision, overseers of the WfD system also need data on the demand for skills and the nature of skills gaps and mismatches. In many developing and emerging economies, educational attainment and self-reported literacy are often the only available measures of “skills.� International cognitive tests provide additional insights but the data typically pertain only to school-age youth; they also ignore the importance of non-cognitive skills for workplace success in the twenty-first century (for example, di Gropello et al. 2010 and 2011; Sondergaard and Murthi 2012). To close the information gap, countries in the Organization for Economic Co-operation and Development (OECD) have embarked on new surveys under the Programme for the International Assessment of Adult Competencies (PIAAC) to document and analyze the skills of the working age population (Schleicher 2008). A similar effort by the World Bank, the Skills toward Employment and Productivity (STEP) Measurement Study, is currently underway in 13 low- and middle-income countries to document and analyze both the cognitive and non-cognitive skills of adults. Finally, assessing promising innovations in WfD is also a desirable aspect of monitoring and evaluation. Mulgan and Albury (2003) postulates that the process of innovation typically passes through four phases: (a) generating possibilities through new ideas; (b) incubating and prototyping the most promising of the new ideas; (c) replicating and scaling up the best ideas (i.e., most cost-effective and most relevant to the local context); and (d) distilling and codifying lessons to institutionalize knowledge and foster a culture of continuous learning and improvement. While many countries lack such a systematic process, a welcome trend is the growing practice of rigorous impact evaluation in the field of skills development. Recent examples include evaluations of youth training programs in several Latin American countries (for example, Ibarraran and Shady 2009; Orazio et al 2011) and of programs targeted at adolescent girls in countries around the world. 55 In light of the ideas discussed above, the SABER-WfD tool assesses the goal of enhancing accountability for results by gathering information on practices relating to monitoring and evaluation, including the following: (a) the reporting of administrative data by training providers; (b) the prevalence and nature of complementary data collection on skills-related topics; and (c) the use of data to monitor and improve the performance of programs and the system as a whole. Summary of the Functional Dimensions and Related Policy Goals In summary, the SABER-WfD tool collects data on three functional dimensions of institutions, policies and praxis for workforce development: strategy, oversight, and service delivery. For each dimension, the data focus attention on policy goals that address market and government failures in governance, finance, and information. Figure 4 recapitulates the policy goals explained above while Appendix 1 provides additional details. and of one of Nanyang Polytechnic, at http://www.nyp.edu.sg/about-nyp/media-centre/publications/annual-report (accessed August 12, 2012). 55 For more details on the Adolescent Girls Initiative at the World Bank see http://go.worldbank.org/ 5PYHEZS360 (accessed on November 9, 2011). 32 Figure 4: Summary of the Functional Dimensions and Policy Goals in the SABER-WfD Dataset Dimension Policy Goal Policy Action • 1. Setting a strategic direction for WfD 4 policy actions Strategy • 2. Prioritizing a demand-led approach • 3. Strengthening critical coordination 10 topics • 4. Ensuring efficiency and equity in funding 9 policy actions Oversight • 5. Assuring relevant and reliable standards • 6. Diversifying pathways for skills acquisition 21 topics • 8. Enabling diversity and excellence in training provision 5 policy actions Delivery • 7. Fostering relevance in public in training programs • 9. Enhancing evidence-based accountability for results 16 topics Source: Authors’ construction. Note: See Appendix 1 for details on the policy actions. Design Features of the SABER-WfD Data Collection Instrument The idiosyncrasy and complexity of WfD systems present challenges for the design of the SABER-WfD data collection instrument (DCI). They make a persuasive case for gathering detailed information on all aspects of the WfD system. Yet the essentially qualitative nature of the data would make gathering such data too time-consuming and costly. Moreover, these data would need to be standardized in order to facilitate self-evaluation and cross-country learning. Below we highlight our approach to addressing some of these challenges and explain the protocols for data collection. Balance between diagnostic and comprehensive data. The SABER-WfD data collection instrument (DCI) is parsimonious by necessity. It thus focuses on key WfD institutions, policies and praxis, such as: the presence of particular policies and their scope; the institutions through which the policies operate; the involvement of and engagement with stakeholders; and evidence of policies being implemented and of feedback loops used to adjust and improve policies and implementation. To standardize information across countries, all the questions in the DCI are answered by choosing from a closed menu of options, and substantiated by documentary evidence or information from credible informants. As appropriate, the data collector may provide additional information to qualify the answer option selected, thus avoiding the loss of potentially important country-specific information. A more difficult challenge relates to the complexity of WfD systems. Such systems typically involve multiple authorities, often with complementary as well as overlapping responsibilities. The training programs that are available may target clients at different levels of instruction, and training providers are likely to serve a diversity of employers from different sectors of the economy. In large countries, the WfD system may be highly differentiated across regions. Under the circumstances, the temptation is to 33 collect information on all aspects of the system. Yet, such comprehensive data may provide too much detail for the diagnostic purpose of the SABER-WfD initiative. As a practical strategy to address the complexity of WfD systems, we assume that WfD institutions, policies and praxis may be benchmarked based on information associated with the system’s most salient or dominant components. This approach puts all systems in the best possible light, thus removing some of the ambiguity in the benchmarking exercise. Unless a single WfD authority oversees both pre- employment and in-service training, the DCI requests information about the ministries most likely to be involved in oversight and service delivery functions; in most cases, the ministry of education and the ministry of labor are the key WfD authorities. To take into account country specificities, the DCI includes an option for the data collector to include other ministries should these play an important role in the system. The DCI addresses complexity in training provision by requesting information separately for two levels of instruction: secondary and post-secondary. For a few questions in the DCI, meaningful answers are possible only in relation to particular economic sectors (for example, regarding the linkages between training provision and industry) or occupations (for example, the organization of skills testing). In such cases, the DCI requests information for the two most dominant or fastest-growing sectors or occupations. For large countries, data collection through the DCI is more feasible and meaningful at the subnational level rather than for the country as a whole. Data on the de jure and the de facto situation. An important concern with the type of qualitative data collected by the DCI is that they may reflect the de jure situation rather than the reality on the ground. For example, many low-income countries have well-written plans to make WfD a key asset for economic development and some even report plans for sophisticated funding systems for WfD (for example, vouchers for individuals to pay for skills development). Yet, for a variety of reasons, including weak capacity and lack of resources, the plans are often not implemented. In such situations, it would be inappropriate to benchmark the system based entirely on the intended policies rather than on actual practice. A full-fledged data collection effort would help verify implementation of the stated plans and policies but may be impractical because of cost and time considerations. To strike a balance, we adopt the strategy of adding a few more questions under each topic of inquiry in the DCI to find out if the most important plans, policies, or programs among those listed, typically no more than the top two, have been implemented, and evaluated for impact, and if the recommendations from the evaluations have led to follow-up actions. The information clearly does not fully capture the de facto situation and it is certainly no substitute for a more comprehensive documentation of the status of policy implementation. The approach nonetheless provides sufficient information to make it possible to situate the policy goal being benchmarked in relation to the four stages of development used in our benchmarking exercise. Analyzing the SABER-WfD dataset The SABER-WfD dataset consists of qualitative information on various aspects of a WfD system. Below we explain how the data are analyzed to obtain scores that rate the maturity of the system’s institutions and policy processes. 34 Generic scoring rubrics. Following the practice adopted in the World Bank’s overall SABER initiative, we use a four-point scale for scoring the data collected by the DCI: latent, emerging, established, and advanced. The generic meaning of the scale appears in Figure 5 while its application to each aspect of WfD institutions, policies and praxis documented by the tool is conditioned by the specific questions posed. Figure 5: Generic Benchmarking Rubrics Advanced Established •Systemic good •Systemic practice good practice meeting Emerging global •Some standards instances of good practice Latent •Limited engagement Source: Authors’ construction. Rubrics for scoring topic level data. The DCI contains questions organized by topic. Although each question generates a data point, the information is more substantive when combined with other information under the same topic. For example, on the first topic in the DCI—“advocacy for WfD to support economic development�—a total of eight questions are posed, to seek such information as: (a) whether or not there are visible champions that advocate for WfD; (b) who these champions are; (c) how the champions have provided advocacy for WfD; and so on. Analyzing the information taken together rather than separately for each question provides a fuller picture of the existence and nature of high-level advocacy for WfD. For this reason, in defining the scoring rubrics for data analysis, we treat the topic as the basic unit of analysis, not the individual questions for each topic. In this sense, the topic-level rubrics might be described as the foundational rubrics in our analytical approach. For each of the 47 topics in the DCI, the rubric is customized by adapting the generic rubric to the specificities of the topic under consideration. As an example, table 1 below shows the rubrics for the first two topics in the DCI. 56 56 See Appendix 2 for additional examples. 35 Table 1: Examples of Customized Rubrics for Scoring Topic-Level Data a/ Scoring Rubrics Topic Latent (1) Emerging (2) Established (3) Advanced (4) Topic 1: No visible champions Some visible Government leaders Both government and Advocacy for advocate for WfD to champions advocate exercise sustained non-government leaders WfD to support economic for WfD to support advocacy for WfD with exercise sustained Support development. economic occasional, ad-hoc advocacy for WfD, and Economic development on an participation from non- rely on routine, Development ad-hoc and limited government leaders; institutionalized basis. their advocacy focuses processes to collaborate on the WfD policy on a strategic, economy- agenda for selected wide WfD policy agenda. industries or economic sectors. Topic 2: WfD champions have WfD champions have WfD champions have WfD champions have Strategic taken no specific taken specific action taken specific action on taken specific action on Focus and action to advance on strategic WfD strategic WfD priorities strategic WfD priorities Decisions by strategic WfD priorities through a through a range of through well-integrated the WfD priorities. few interventions, interventions, and interventions, and Champions but no arrangements implementation progress implementation progress exist to monitor and is monitored, albeit is monitored and review through ad-hoc reviews. reviewed through implementation routine, institutionalized progress. processes. a/ Numbers in parentheses correspond to the scores assigned to each point on the rubrics used to analyze the SABER-WfD data. Source: Authors’ construction. The data for each topic in the DCI are scored against the rubric for that topic, using a four-point scale. The scores are calibrated based on available knowledge on global good practice and reflect the principles reflected in the generic rubrics. The qualitative nature of the data and the fact that multiple data points are involved mean that a certain amount of ambiguity is unavoidable in the scoring process. We minimize the problem in several ways. One approach is to identify for each topic, a few critical conditions among all the ones listed that must be satisfied in order to earn a given score; failing to meet these critical conditions automatically pushes the score to the next lower level even if all other conditions for that score are satisfied. The choice of these critical conditions differs by topic, but they would often pertain to the scope of policies or engagement with relevant stakeholders, or to the degree of institutionalization of key policy processes. Other measures to minimize ambiguity in the scores involve strengthening the scoring process itself. At the start of the process, a typical approach is to arrange for separate scoring of the data by several individuals, all using the same scoring rubric. Their scores are then compared and either confirmed or rationalized to remove inconsistencies following discussion among the scorers and, if needed, between them and those responsible for filling in the DCI. A process of consultation with knowledgeable in- country stakeholders offers yet another measure to remove ambiguity and validate the scores. The 36 main goal of all these approaches is to ensure that the scores provide a sufficiently accurate picture of the state of the WfD system’s institutions, policies and praxis. Aggregate scores and “composite� rubrics. In the SABER-WfD analytical framework, the topics feed into higher levels of aggregation, first to the level of policy actions, then to the level of policy goals, and finally to the level of functional dimensions. This structure simplifies the complexity of WfD institutions, policies and praxis to more manageable components. It also means that the topic-level scores can be used to compute scores for the higher levels of aggregation in the framework. Scores at the policy action level are thus simply the unweighted averages of the underlying topic-level scores; those at the policy goal level are the unweighted averages of the underlying action-level scores; and those at the dimension level are the unweighted averages of the scores for the underlying policy goals. This algorithm yields scores on a 1-4 scale for every level of aggregation in the data. However, since the composite scores are averages of the underlying foundational scores, they are rarely whole numbers. 57 To interpret the aggregate scores we develop composite rubrics which are based on the topic-level rubrics. These composites are almost always less precise than the rubrics on which they are based. A key reason is the inevitable increase in the number of permutations of conditions, so that the same score may be consistent with many combinations of underlying conditions at the topic level. Yet by reducing the large amount of detailed information at the topic level to a digestible volume, the composite rubrics remain meaningful for assessing the maturity of WfD institutions, policies and praxis. To take the example of the two topics in the previous table, the composite rubric for the policy action into which they feed (viz. “provide sustained advocacy for WfD at the top leadership level�) is shown in table 2 below. 58 Table 2: Example of a policy action-level composite rubric a/ Latent (1) Emerging (2) Established (3) Advanced (4) Policy Action: Visible champions for Some visible champions Government leaders Both government and provide WfD are either absent or provide ad-hoc advocacy exercise sustained non-government leaders sustained take no specific action for WfD and have acted advocacy for WfD with exercise sustained advocacy for to advance strategic on few interventions to occasional, ad-hoc advocacy for WfD, and WfD at the top WfD priorities. advance strategic WfD participation from non- rely on routine, leadership level priorities; no government leaders; their institutionalized arrangements exist to advocacy focuses on processes to collaborate monitor and review selected industries or on well-integrated implementation progress. economic sectors and interventions to advance a Note: this manifests itself through a strategic, economy-wide action is a range of specific WfD policy agenda; consolidation interventions; implementation progress 57In order to conform to standardized presentation across all policy domains in the overall SABER initiative, the composite SABER-WfD scores may sometimes need to be converted to whole numbers mapped to a generic rubric with a four-point scale corresponding to latent, emerging, established and advanced. For this purpose, the following conversion table is used: Composite score 1.00 ≤ X ≤ 1.75 1.75 < X ≤ 2.50 2.50 < X ≤ 3.25 3.25 < X ≤ 4.00 Generic rubric and Latent Emerging Established Advanced score 1 2 2 4 58 See Appendix 3 for the composite rubrics for all 18 policy actions in the SABER-WfD framework. 37 of the two implementation progress is is monitored and topics in the monitored, albeit through reviewed through routine, previous table ad-hoc reviews. institutionalized processes. a/ This composite rubric is a consolidation of the rubrics shown in the previous table for the two topics which feed into the policy action of “provide sustained advocacy for WfD at the top leadership level. The bolded words draw the reader’s attention to key differences across the scores defined by the rubric. Source: Authors’ construction. The scores at the policy action level contain rich information but they also risk slicing reality too finely and preventing a more holistic assessment to emerge from the data. Scores at the next higher levels of aggregation at the policy goal and policy dimension levels address this concern. However, the inevitable increase in the number of permutations that fit a particular score implies a corresponding loss of precision in the composite rubrics. As a compromise between precision and brevity in reporting the findings, SABER-WfD reports would generally present the scores aggregated at the policy goal and dimension levels. The rubrics for the nine policy goals appear in Appendix 4. One final note on aggregated scores is noteworthy. As the reader may recall, the DCI does not collect comprehensive data. Yet where the WfD system involves more than one government ministry or more than one level of instruction, it gathers data separately for each relevant ministry or agency and for each level of instruction. For such data, the topic-level information is first scored separately by ministry and by level of instruction, based on the relevant rubric for that topic; these scores are then aggregated, again by simple averages, to obtain the composite topic level score. The aggregation to higher levels of aggregation follow the same procedure as described above. The scoring algorithm has been kept intentionally simple. The straightforward calculations have an important advantage: they allow easy unpacking of each score to reveal the underlying topic-level scores; they also mean that scores can be disaggregated, if needed, by agency and level of instruction. These features make it relatively easy to explore the reasons behind a high or low score, thus drawing attention to specific areas of strength or potential weaknesses in the system’s institutions or policy processes. 38 Conclusion Workforce development is a topic of growing interest among the World Bank’s partner countries. Many policy makers in these countries acknowledge that skills gaps and mismatches can potentially cripple efforts to boost economic growth and reduce poverty. Yet it is unclear what specific approaches and options for skills development might help advance desired socioeconomic goals. Given the cross-cutting nature of WfD challenges, policy makers need an appropriate analytical framework for clarifying and facilitating dialogue on these challenges. They also need information on the how the WfD system is functioning and performing, not only in their own countries, but also elsewhere, particularly in countries that are making progress in equipping the workforce with job-relevant and productivity-enhancing skills. The SABER-WfD package of resources was created to meet both of these needs. The package contains an analytical framework that brings into focus both the supply and demand sides of the skills equation. Accompanying the framework is a data collection instrument, now in version 2.5, which gathers data on key aspects of WfD institutions, policies and praxis. This type of information complements other data, such as those from the World Bank’s STEP Skills Measurement study and the OECD’s PIAAC initiative, which provide focus on measuring the skills of adults across countries, on documenting the nature of gaps and mismatches, and on exploring the links between skills, jobs and productivity. The SABER-WfD tool focuses instead on the mechanisms through which government interventions might influence outcomes in workforce development. Its niche is to systematize the collection and analysis of typically qualitative information on WfD institutions, policies and praxis. Experience thus far with the SABER-WfD tool is promising. Its analytical framework is helping to draw attention to the importance of taking a cross-sector and holistic approach to addressing WfD challenges. The systematic documentation of WfD institutions and policies is shedding light on the inner workings of WfD systems, a feature that often receives limited or only ad-hoc attention in analytical work. By enabling the accumulation of comparable data across countries on these aspects of WfD systems, the SABER-WfD tool is also enabling the accumulation of cross-country information on promising approaches to strengthen WfD systems. These benefits make the tool an important component of the World Bank’s implementation strategy for its Education Strategy 2020. While it is premature to evaluate the tool’s impact, the early results are positive and encourage its use to support and stimulate dialogue on the design and implementation of WfD policies. 39 References Abe, Masaki. 2006. “The Developmental State and Educational Advance in East Asia.� Educate 6(1): 6-12 Abrams, Peter, Grazia Borrini-Feyerabend, Julia Gardner, and Pippa Heylings. 2003. “Evaluating Governance: A Handbook to Accompany a Participatory Process for a Protected Area.� Unpublished manuscript. Gatineau, Quebec: Parks Canada and TILCEPA. http://cmsdata.iucn.org/downloads/ evaluating_governance_handbook.doc (accessed on October 27, 2011). Adams, Arvil V. 2007. “The Role of Youth Skills Development in the Transition to Work: A Global Review.� HDNCY 5, Human Development Network Children and Youth Department, World Bank, Washington, DC. http://siteresources.worldbank.org/INTCY/Resources/395766- 1187899515414/RoleofYouthSkills.pdf (accessed on October 27, 2011). Allais, Stephanie. 2010. The Implementation and Impact of National Qualifications Frameworks: Report of a Study in 16 countries. Geneva: Skills and Employability Department, International Labour Office (ILO). http://www.ilo.org/wcmsp5/groups/public/---ed_emp/---ifp_skills/ documents/meetingdocument/wcms_126589.pdf (accessed on October 27, 2011). Almeida, Rita, Jere Behrman, and David Robalino. 2012. “The Economic Rationale for Skills Development Policies.� In The Right Skills for the Job? Rethinking Training Policies for Workers, ed. Rita Almeida, Jere Behrman, and David Robalino. Washington, DC: Human Development Perspectives, World Bank. http://elibrary.worldbank.org/content/book/9780821387146 (accessed on August 13, 2012). Alstete, Jeffrey W. 1995. “Benchmarking in Higher Education.� ASHE-ERIC: Higher Education Report 5. The George Washington University Graduate School of Education and Human Development, Washington DC. Ahston, David. 2006. “Lessons from Abroad: Developing Sector Based Approaches to Skills,� SSDA Catalyst, Issue 2. London: Skills for Business, Sector Skills Development Agency (SSDA). Anderson, Kym. 2010. “Contributions of the Innovation System to Australia’s Wine Industry Growth.� WERC Working Paper 0310. Wine Economics Research Centre, University of Adelaide, Adelaide. Ashton, David, Francis Green, Johnny Sung, and Donna James. 2002. “The Evolution of Education and Training Strategies in Singapore, Taiwan and S. Korea: A Development Model of Skill Formation.� Journal of Education and Work 15(1): 5–30. Asian Development Bank (ADB). 2009. Good Practice in Technical and Vocational Education and Training. Manila: Asian Development Bank. http://www.adb.org/ Documents/Guidelines/Good- Practice-Education-Training/Good-Practice-Education-Training.pdf (accessed on October 27, 2011). ———. 2010. Education and Skills: Strategies for Accelerated Development in Asia and the Pacific. Mandaluyong City, Philippines: Asian Development Bank. Atchoarena, David. 2002. “Private Training.� In World Bank Study on: Vocational Skills Development in Sub-Saharan Africa—a Working Group Review, ed. Martha Caddell and Keith Holmes. Edinburgh: Working Group for International Cooperation in Skills Development. http://www.norrag.org/ medias/documents/pdf/publications/papers/Paper7.pdf (accessed on October 27, 2011). 40 Autor, David. 2010. The Polarization of Job Opportunities in the U.S. Labor Market: Implications for Employment and Earning. Washington, DC: The Center for American Progress and The Hamilton Project. Autor, David, Frank Levy, and Richard J. Murnane. 2003. “The Skills Content of Recent Technological Change: An Empirical Exploration.� Quarterly Journal of Economics 118 (4): 1279-1333. Banerji, Arup, Robert Holzmann, Pierella Paci, Carmen Pages, Stefano Scarpetta, and Milan Vodopivec. 2008. “MILES to Go: A Quest for an Operational Labor Market Paradigm for Developing Countries.� Unpublished manuscript. Social Protection and Labor Sector, World Bank, Washington, DC. http://siteresources.worldbank.org/INTLM/Resources/390041-1212776476091/5078455- 1267646113835/MILESThequestoperational LMparadigm_Jan212008.pdf (accessed on October 27, 2011). Betcherman, Gordon, Karina Olivas, and Amit Dar. 2004. “Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries.� Social Protection Discussion Paper Series 0402, World Bank, Washington, DC. http://info.worldbank.org/etools/docs/library/251019/day6DiscussionPaperSeries0402April6Se1. pdf (accessed on October 27, 2011). Betcherman, Gordon, Martin Godfrey, Olga Susana Puerto, Friederike Rother, and Antoneta Stavreska. 2007. “Global Inventory of Interventions to Support Young Workers: Synthesis Report.� Social Protection Discussion Paper 0715, World Bank, Washington, DC. http://siteresources. worldbank.org/EXTECAREGTOPYOUTH/Resources/YEI_Synthesis_Paper_Final_July_13_2007.pdf (accessed on October 27, 2011). Blom, Andreas, and Hiroshi Saeki. 2011. “Employability and Skill Set of Newly Graduated Engineers in India.� World Bank Policy Research Working Paper No. 5640, World Bank, Washington, DC. Bloom, Benjamin.S. 1976. Human Characteristics and School Learning. New York: McGraw-Hill. Booth, Alison L., and Dennis J. Snower. 1996. “Introduction: Does the Free Market Produce Enough Skills?� In Acquiring Skills: Market Failure, Their Symptoms and Policy Responses. Alison L. Booth and Dennis J. Snower, eds. Cambridge: Press Syndicate of the University of Cambridge. Bosch, Gerhard. 2010. The Revitalization of the Dual System of Vocational Training in Germany. New York: Routledge Camp, Robert C. 1989. Benchmarking: The Search for Industry Best Practices that Lead to Superior Performance. Milwaukee, Wisconsin: ASQC Quality Press. Center for the Development of Vocational Training (CEDEFOP). 2010. Skills Supply and Demand in Europe. Medium-Term Forecast Up To 2020. Luxembourg: Publications Office of the European Union. http://www.cedefop.europa.eu/download-manager.aspx?id=15535&lang= en&type=publication (accessed on October 27, 2011). Commission on Growth and Development. 2008. The Growth Report: Strategies for Sustained Growth and Inclusive Development. Washington, DC: World Bank. http://cgd.s3.amazonaws.com/GrowthReportComplete.pdf (accessed on October 27, 2011). Commission on Growth and Development. 2010. Leadership and Growth. Washington, DC: World Bank Dattakumar, R., and R. Jagadeesh. 2003. “A Review of Literature on Benchmarking.� Benchmarking: An International Journal 10(3): 176–209. 41 de Largentaye, Armand Rioust. 2009. “Vocational Training and the Informal Economy.� In Promoting Pro-Poor Growth Employment. Paris: OECD. http://www.oecd.org/dataoecd/63/11/43514554.pdf (accessed on October 27, 2011). Deloitte, The Manufacturing Institute, and Oracle (DMIO). 2009. “People and Profitability: A Time for Change: A 2009 People Management Practices Survey of the Manufacturing Industry.� http://www.deloitte.com/ assets/Dcom-UnitedStates/Local%20Assets/Documents/us_pip_people managementreport_100509.pdf (accessed on October 27, 2011). di Gropello, Emanuela, with Hong Tan and Prateek Tandon. 2010. Skills for the Labor Market in the Philippines. Washington, DC: World Bank. di Gropello, Emanuela, with Aurelien Kruse and Prateek Tandon. 2011. Skills for the Labor Market in Indonesia: Trends in Demand, Gaps, and Supply. Washington, DC: World Bank. http://ddp- ext.worldbank.org/EdStats/ IDNpub11.pdf (accessed on October 27, 2011). Dunbar, Muriel. 2011. “Engaging the Private Sector in Skills Development.� Guidance Notes: A DFID Practice Paper. London: Department for International Development, Government of the United Kingdom. Elkins, Jill, Chris Krezeminski, and Carl Nink. 2012. Labor Market Analysis Leads to Demand-Driven TVET Programs. Centreville, Utah: Management and Training Corporation. European Training Foundation (ETF). 2012. The Torino Process, Turin: ETF, at http://www.etf.europa.eu/web.nsf/pages/TORINO_PROCESS_2012 (accessed on 12/26/2012) Federal Office for Professional Education and Technology (FOPET). 2012. Vocational and Professional Education and Training in Switzerland: Facts and Figures. Basel: Federal Office for Professional Education and Technology. Ferrier, Fran, Clifford Trood, and Karen Whittingham. 2003a. Going Boldly into the Future: A VET Journey into the National Innovation System, Adelaide: National Centre for Vocational Education Research (NCVER), Australian National Training Authority. http://www.ncver.edu.au/research/proj/nr9036_ vol1.pdf (accessed on October 27, 2011). Ferrier, Fran, Clifford Trood, and Karen Whittingham. 2003b. Going Boldly into the Future: A Series of Case Studies of Co-operative Research Centres and their Relationships with the VET Sector. Adelaide: National Centre for Vocational Education Research (NCVER), Australian National Training Authority. http://www.ncver.edu.au/research/proj/nr9036cs.pdf (accessed on October 27, 2011). Finegold, David. 1995. Workforce Development in the United States: Creating a More Effective Training Market. Santa Monica: Rand Corporation. Ganzglass, Evelyn, Keith Bird, and Heath Prince. 2011. Giving Credit Where Credit is Due: Creating a Competency-Based Qualifications Framework for Postsecondary Education and Training. Washington, DC: Center for Postsecondary and Economic Success. http://www.clasp.org/admin/site/publications/files/ Giving-Credit.pdf (accessed on October 27, 2011). Gereffi, Gary, Karina Fernandez-Stark, and Phil Psilos. 2011. Skills for Upgrading: Workforce Development and Global Value Chains in Developing Countries. Durham, North Carolina: Center on Globalization, Governance and Competitiveness. 42 Green, Francis, David Ashton, Donna James, and Johnny Sung. 1999. “The Role of the State in Skill Formation: Evidence from the Republic of Korea, Singapore, and Taiwan.� Oxford Review of Economic Policy 15(1): 82–96. Hanf, Georg, and Ute Hippach-Schneider. 2005. “What Purpose do National Qualifications Frameworks Serve? A look at Other Countries.� Vocational Training in Research and Practice (Bundesinstitut für Berufsbildung) Special Edition H 20155 (Promoting Innovations in Vocational Education and Training: An Exchange of German Experiences): 9–14. http://www.bibb.de/ veroeffentlichungen/en/publication/download/id/1692 (accessed on October 27, 2011). Hanushek. Eric A., and Ludger Wößmann. 2006. “Does Educational Tracking Affect Performance and Inequality? Differences-in-Differences Evidence Across Countries.� Economic Journal 116(510): C63–C76. Heckman, James J., Jora Stixrud, and Sergio Urzua. 2006. “The Effects of Cognitive and Noncognitive Abilities on Labor Market Outcomes and Social Behavior.� NBER Working Paper 12006, National Bureau of Economic Research, Cambridge, MA. Hicks, Joan Hamory, Michael Kremer, Isaac Mbiti, and Edward Miguel. 2011. “Vocational Education Voucher Delivery and Labor Market Returns: A Randomized Evaluation among Kenyan Youth.� Report for Spanish Impact Evaluation Fund (SIEF) Phase II, World Bank, Washington, DC. Government of India Planning Commission, 2012. Drat Twelfth Five Year Plan 2012-17, Volume 3. New Delhi: Planning Commission, Government of India. http://planningcommission.gov.in/plans/planrel/12thplan/pdf/vol_3.pdf (accessed January 7, 2013). Ibarraran, Pablo, and David Rosas Shady. 2009. “Evaluating the Impact of Job Training Programs in Latin America: Evidence from IDB Funded Operations.� Journal of Development Effectiveness 2(1): 195– 216. International Labour Office (ILO). 2008. Skills for Improved Productivity, Employment Growth and Development, Report V. International Labour Conference, 97th Session. Geneva: International Labour Office. International Labour Office (ILO). 2010. “Teachers and Trainers for the Future – Technical and Vocational Education and Training in a Changing World.� Report for discussion at the Global Dialogue Forum on Vocational Education and Training, Sectoral Activities Programme, ILO, Geneva. http://www.ilo.org/wcmsp5/groups/public/@ed_dialogue/@sector/documents/meetingdocume nt/wcms_161661.pdf (accessed on January 8, 2013). Jacobs, Ronald L., and Joshua Hawley. 2008. “Emergence of Workforce Development: Definition, Conceptual Boundaries, and Implications.� In International Handbook of Education for the Changing World of Work (Vol. 6), ed. R. MacLean and D. Wilson. Dordrecht, Netherlands: Springer Science + Business Media B.V. Janjua, Shehryar, and Arif Naveed. 2009. “Skills Acquisition and the Significance of Informal Training System in Pakistan—Some Policy Implications.� RECOUP Policy Brief 7, Research Consortium on Educational Outcomes and Poverty, Cambridge. http://www.dfid.gov.uk/r4d/PDF/ Outputs/ImpOutcomes_RPC/pb7.pdf (accessed on October 29, 2011). 43 Johanson, Richard. 2009. “A Review of National Training Funds.� Social Protection Discussion Paper 0922, World Bank, Washington, DC. http://siteresources.worldbank.org/SOCIALPROTECTION/Resources/SP-Discussion-papers/Labor- Market-DP/0922.pdf (accessed on October 29, 2011). Johanson, Richard, and Arvil Van Adams. 2004. Skills Development in Sub-Saharan Africa. World Bank Regional and Sectoral Studies. Washington, DC: World Bank. Karmel, Tom. 2011. The Challenge of Measurement: Statistics for Planning Human Resource Development. Adelaide: National Centre for Vocational Education Research. http://www.ncver.edu.au/research/core/2418.pdf (accessed on October 29, 2011). Kiley, Margaret, and Robert Cannon. 2000. Leap into Lifelong Learning. Adelaide: Centre for Learning and Professional Development, University of Adelaide. King, Kenneth, and Robert Palmer. 2010. Planning for Technical and Vocational Skills Development. UNESCO: International Institute for Educational Planning. Korea Research Institute for Vocational Education and Training (KRIVET). 2007. Human Resources Development Indicators in Korea, 2007. Seoul: KRIVET. Kuruvilla, S., C. L. Erickson, and A. Hwang. 2002. “An Assessment of the Singapore Development System: Does It Constitute a Viable Model for Other Developing Countries?� World Development 30(8): 1461–1476. Law, Song Seng. 2008. “Vocational Technical Education and Economic Development: The Singapore Experience.� In Toward a Better Future. Education and Training for Economic Development in Singapore since 1965, eds. Lee Sing Kong, Goh Chor Boon, Birger Fredriksen, and Tan Jee Peng. Washington, DC: World Bank and National Institute of Education, Singapore. Law, Song Seng, 2013 (forthcoming). A Breakthrough in Vocational and Technical Education. - The Singapore Story, Institute of Policy Studies, Singapore. Lee, Young-Hyun. 2009. “Vocational Education and Training in the Process of Industrialization.� Understanding Korean Educational Policy, Vol. 5. Educational Development Institute, Seoul. Korean. http://eng.kedi.re.kr/ khome/eng/education/educationSeries.do# (accessed on October 30, 2011). Levy, Santiago. 2008. Good Intentions, Bad Outcomes: Social Policy, Informality, and Economic Growth in Mexico. Washington, DC: Brookings Institution Press. Levy, Frank, and Richard J. Murnane. 2005. The New Division of Labor: How Computers Are Creating the Next Job Market. Princeton, NJ: Princeton University Press. Lindburg, Lori. 2009. Replicating Success: Innovative Collaborations between the Biotech Industry and Education. San Francisco: Bay Bio Institute, City College of San Francisco. http://www.bio- link.org/home/sites/files/replicating success_finalscreen.pdf (accessed on October 30, 2011). Orazio, Attanasio, Adriana Debora Kugler, and Costas Meghir. 2011. “Subsidizing Vocational Training for Disadvantaged Youth in Columbia: Evidence from a Randomized Trial.� American Economic Journal: Applied Economics. 3(3): 188-220. MacKenzie, Jeanne, and Rose-Anne Polvere. 2009. “TVET Glossary: Some Key Terms.� In International Handbook of Education for the Changing World of Work, ed. R. Maclean and D. Wilson. Dordrecht, Netherlands: Springer Science + Business Media B.V. 44 Memedovic, Olga. 2004. Inserting Local Industries into Global Value Chains and Global Production Networks: Opportunities and Challenges for Upgrading. Vienna: United Nations Industrial Development Organization. Middleton, John, Adrian Ziderman, and Arvil Van Adams. 1993. Skills for Productivity: Vocational Education and Training in Developing Countries. Washington, DC: World Bank. Milliron, Mark David. 2007. “Transcendence and Globalization: Our Education and Workforce Development Challenge.� New Directions for Community Colleges 2007(138): 31-38. Mupinga, Davison M., Joe R. Busby, and Joseph W. Ngatiah. 2006. “Postsecondary Technical and Vocational Education Institutions in Kenya: Needs and Challenges.� International Journal of Vocational Education and Training 14(1): 21-36. Mourshed, Mona, Chinezi Chijioke, and Michael Barber. 2010. How the World’s Most Improved School Systems Keep Getting Better. McKinsey and Company. http://mckinseyonsociety.com/downloads/reports/Education/How-the-Worlds-Most-Improved- School-Systems-Keep-Getting-Better_Download-version_Final.pdf (accessed on October 30, 2011). Mulgan, Geoff, and David Albury. 2003. Innovation in the Public Sector. London: Strategy Unit, Cabinet Office. O’Hare, Daniel. 2008. “Education in Ireland: Evolution of Economic and Education Policies since the Early 1990s.� In An African Exploration of the East Asian Education Experience, ed. Birger Fredriksen and Tan Jee Peng. Washington, DC: World Bank. http://www.adeanet.org/adeaPortal/ adea/Biennale%202008/Documentation/Papers%20for%20presentation/01.%20Session%201/Fin al%20PDF%20documents/Session%201%20Doc%204%20World%20Bank%20ENG.pdf (accessed on October 30, 2011). O'Lawrence, Henry. 2007. “A Case Study Report of a Comprehensive Benchmarking Review of the Status of Vocational Education Students in California: Building a New Relationship between Community Colleges and California State University System.� Journal of Career and Technical Education 23(1): 85–96. http://scholar.lib.vt.edu/ejournals/JCTE/v23n1/pdf/ olawrence.pdf (accessed on October 30, 2011). Organization for Economic Cooperation and Development (OECD). 2010a. Learning for Jobs: Synthesis Report of the OECD Reviews of Vocational Education and Training. Paris: OECD http://www.mskills.com.au/DownloadManager/ downloads/Learning%20for%20jobs%20report.pdf (accessed on October 30, 2011). ———. 2010b. PISA 2009 Results: What Students Know and Can Do: Student Performance in Reading, Mathematics and Science. Vol. 1. Paris: OECD. http://www.oecd.org/dataoecd/10/61/48852548.pdf (accessed on October 30, 2011). Oster, Emily, and M. Bryce Millett. 2010. “Do Call Centers Promote School Enrollment? Evidence from India.� NBER Working Paper 15922, National Bureau of Economic Research, Washington, DC. Prasad, Sharda (2012) “Skill Development: Indian Experience--Policies and Programmes,� presentation at the World Bank South Asia Regional Experience Sharing Workshop, June 25, 2012. Rodgers, F. Halsey (2012). Getting to Know SABER: The What, Why, and How of the Systems Approach for Better Education Results, Education Department, Human Development Network, World Bank (draft).. 45 Schleicher, Andreas. 2008. “PIAAC: A New Strategy for Assessing Adult Competencies.� International Review of Education 54(5-6): 627-650. Schumpeter, Joseph A. 2003. Capitalism, Socialism and Democracy. Taylor & Francis e-Library edition. London: George Allen & Unwin. Singapore Accreditation Council. 2008. “Frequently Asked Questions: What Are the Benefits of Being Accredited?� http://www.sac-accreditation.gov.sg/ accreditation.asp#qn1 (accessed on June 17, 2011). Singh, Chandandeep, and Kuldeep Sareen. 2006. “Effectiveness of ISO 9000 Standards in Indian Educational Institutions: A Survey.� International Journal of Services Technology and Management 7(4): 403–415. Sliwka, Anne. 2010. From homogeneity to diversity in German education,� in Educating Teachers for Diversity. Meeting the Challenge. Centre for Educational Research and Innovation. Paris: Organization for Economic Cooperation and Development Skills Victoria. 2011. “Victorian Government Response to Skills Australia Consultation: Creating a Future Direction for Australian Vocational Education and Training.� Mimeo. Skills Victoria, Department of Education and Early Childhood Development, Melbourne. http://www.eduweb. vic.gov.au/edulibrary/public/commrel/about/skills/victoriangovernmentresponsetoskillsaustraliac onsultation.doc (accessed on October 30, 2011). Sondergaard, Lars, and Mamta Murthi. 2012. Skills, Not Just Diplomas: Managing Education for Results in Eastern Europe and Central Asia. Directions in Development Series. Washington, DC: World Bank. Spar, Debora. 1998. “Attracting High Technology Investment: Intel’s Costa Rican Plant.� Foreign Investment Advisory Service. Occasional Paper 11, International Finance Corporation and World Bank. Sudan, Randeep, Seth Ayers, Philippe Dongier, Arturo Muente-Kunigami, and Christine Zhen-Wei Qiang. 2010. The Global Opportunity in IT-Based Services. Assessing and Enhancing Country Competitiveness. Washington, DC: World Bank. http://www.infodev.org/en/Document.875.html (accessed on October 30, 2011). Tan, Hong. 2001. “Malaysia’s Human Resource Development Fund: An Evaluation of Its Effects on Training and Productivity.� Mimeo. Washington, DC: World Bank. http://info.worldbank.org/etools/docs/ library/206512/Malaysia%20Human%20Resources%20Development%20Fund%20English.pdf (accessed on October 30, 2011). Tan, Hong, Gladys Lopez-Acevedo, M. Sanchez, and M. Tinajero. 2004. Evaluating SME Programs in Mexico Using Panel Firm Data. Washington, DC: World Bank Institute, World Bank. Tan, Jee-Peng, and Joy Yoo-Jeung Nam. 2012. “Pre-Employment Technical and Vocational Education and Training: Fostering Relevance, Effectiveness, and Efficiency.“ In The Right Skills for the Job? Rethinking Training Policies for Workers, ed. Rita Almeida, Jere Behrman, and David Robalino. Washington, DC: Human Development Perspectives, World Bank http://elibrary.worldbank.org/content/book/9780821387146 (accessed on August 13, 2012). 46 Toze, Melinda and Samantha Tiemey. 2009. Keeping it Real: Industry Currency of Trainers in Queensland. Brisbane� Department of Education and Training, State of Queensland. http://www.vetpd.qld.gov.au/resources/pdf/ic/keeping-it-real.pdf (accessed on January 8, 2013). United Kingdom Commission for Employment and Skills (UKCES). 2009. Ambition 2020: World Class Skills and Jobs for the UK - The 2009 Report. London: UKCES. http://www.ukces.org.uk/assets/bispartners/ukces/docs/ publications/ambition- 2020-the-2009-report.pdf (accessed on November 1, 2011). United Nations Educational, Scientific and Cultural Organization (UNESCO). 2002. “Best Practices� in Technical and Vocational Education: China, the People’s Democratic Republic of Korea and Mongolia. Beijing: UNESCO. http://unesdoc.unesco.org/images/0012/001257/125705eo.pdf (accessed on November 1, 2011). United Nations Population Division of the Department of Economic and Social Affairs (UN). 2009. World Population Prospects: the 2008 Revision. New York: United Nations. Vlăsceanu, Lazăr, Laura Grünberg, and Dan Pârlea. 2007. Quality Assurance and Accreditation: A Glossary of Basic Terms and Definitions. Revised and updated edition. Bucharest: UNESCO-CEPES. http://www.cepes.ro/publications/ pdf/Glossary_2nd.pdf (accessed on November 1, 2011). Wang, Yidan. 2012. Education in a Changing World: Flexibility, Skills and Employability. Washington, DC: Human Development Network, World Bank. (Wolf, Alison. 2011. Review of Vocational Education – The Wolf Report. London: Department for Education. https://www.education.gov.uk/publications/standard/Download?DownloadPublicationReference =DFE-00031-2011&DownloadItemReference=Review%20of%20Vocational%20Education%20- %20The%20Wolf%20Report%20PDF(DfES%20Online%20Store)&DocumentType=PDF&Url=%2Fpu blications%2FeOrderingDownload%2FThe%20Wolf%20Report.pdf (accessed on January 7, 2013). Wolf, Artjom, and Andrea Erdle. 2009. Key Aspects of Economics of Technical and Vocational Education and Training (TEVET): Lessons Learned and Gaps to be Filled. Eschborn: Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ). http://www2.gtz.de/dokumente/bib-2009/gtz2009-0390en- economics-tvet.pdf (accessed on November 1, 2011). Woods, James F., and Christopher J. O’Leary. 2007. “Principles of Labor Market Information.� Employment Research Newsletter 14(2). W.E. Upjohn Institute for Employment Research, Kalamazoo, MI. World Bank. 2003. Lifelong Learning in the Global Knowledge Economy: Challenges for Developing Countries. Washington, DC: World Bank. ———. 2010a. Stepping Up Skills for More Jobs and Higher Productivity. Washington, DC: Human Development Network, World Bank. http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726- 1281723119684/Stepping_up_skills.pdf (accessed on November 1, 2011). ———. 2010b. “Sustaining Educational and Economic Momentum in Africa.� Africa Human Development Series, World Bank Working Paper Series 195, World Bank, Washington, DC. http://siteresources.worldbank.org/ EDUCATION/Resources/278200- 1099079877269/Sustaining_edu_econ_ momentum_Africa.pdf (accessed on November 1, 2011). 47 ——. 2011. Learning for All. Investing in People’s Knowledge and Skills to Promote Development. World Bank Group Education Strategy 2020. Washington, DC: World Bank. http://siteresources.worldbank.org/EDUCATION/Resources/ ESSU/Education_Strategy_4_12_2011.pdf (accessed on November 9, 2011). ——. 2012a. “SABER Overview Paper: The What, Why, and How of the Systems Approach for Better Education Results.� Review draft. Education Department, Human Development Network, World Bank, Washington, DC. ——. 2012b. World Development Report 2013: Jobs. Washington, DC: World Bank. http://siteresources.worldbank.org/EXTNWDR2013/Resources/8258024- 1320950747192/8260293-1322665883147/WDR_2013_Report.pdf (accessed on January 4, 2013).World Bank/Multilateral Investment Guarantee Agency (MIGA). 2006. The Impact of Intel in Costa Rica. Nine Years After the Decision to Invest. Investing in Development Series. Washington, DC: MIGA. http://siteresources. worldbank.org/EXTEXPCOMNET/Resources/2463593- 1213887855468/44_ The_impact_of_Intel_in_Costa_Rica.pdf (accessed on November 1, 2011). Yusuf, Shahid, and Kaoru Nabeshima. 2010. Changing the Industrial Geography in Asia: The Impact of China and India. Washington, DC: World Bank. 48 Appendix 1: Functional Dimensions, Policy Goals, and Actions in SABER-WfD Policy Goal Policy Action Topic in DCI 2.5 Setting a T1 Advocacy for WfD to Support Economic Development Provide sustained advocacy for WfD at the top G1 Strategic leadership level T2 Strategic Focus and Decisions by the WfD Champions Direction Strategic Framework Establish clarity on the demand for skills and T1 Overall Assessment of Economic Prospects and Skills Implications areas of critical constraint Dimension 1 Fostering a T2 Critical Skills Constraints in Priority Economic Sectors G2 Demand-Driven T3 Role of Employers and Industry Approach Engage employers in setting WfD priorities T4 Skills-Upgrading Incentives for Employers and in enhancing skills-upgrading for workers T5 Monitoring of the Incentive Programs Strengthening T1 Roles of Government Ministries and Agencies Formalize key WfD roles for coordinated G3 Critical T2 Roles of Non-Government WfD Stakeholders action on strategic priorities Coordination T3 Coordination for the Implementation of Strategic WfD Measures T1 Overview of Funding for WfD Provide stable funding for effective programs T2 Recurrent Funding for Initial Vocational Education and Training (IVET) in initial, continuing and targeted vocational Recurrent Funding for Continuing Vocational Education and Training Programs education and training T3 Ensuring (CVET) G4 Efficiency and T4 Recurrent Funding for Training-related Active Labor Market Programs (ALMPs) Equity in Funding Monitor and enhance equity in funding for T5 Equity in Funding for Training Programs training Facilitate sustained partnerships between T6 Partnerships between Training Providers and Employers training institutions and employers Broaden the scope of competency standards T1 Competency Standards and National Qualifications Frameworks as a basis for developing qualifications frameworks T2 Competency Standards for Major Occupations System Oversight T3 Occupational Skills Testing Dimension 2 Assuring Establish protocols for assuring the credibility T4 Skills Testing and Certification Relevant and of skills testing and certification G5 T5 Skills Testing for Major Occupations Reliable Standards T6 Government Oversight of Accreditation Develop and enforce accreditation standards T7 Establishment of Accreditation Standards for maintaining the quality of training provision T8 Accreditation Requirements and Enforcement of Accreditation Standards T9 Incentives and Support for Accreditation Promote educational progression and T1 Learning Pathways permeability through multiple pathways, including for TVET students T2 Public Perception of Pathways for TVET Diversifying Facilitate life-long learning through T3 Articulation of Skills Certification G6 Pathways for articulation of skills certification and Skills Acquisition recognition of prior learning T4 Recognition of Prior Learning Provide support services for skills acquisition T5 Support for Further Occupational and Career Development by workers, job-seekers and the disadvantaged T6 Training-related Provision of Services for the Disadvantaged T1 Scope and Formality of Non-State Training Provision Encourage and regulate non-state provision of T2 Incentives for Non-State Providers Enabling Diversity and training T3 Quality Assurance of Non-State Training Provision G7 Excellence in T4 Review of Policies towards Non-State Training Provision Training T5 Targets and Incentives for Public Training Institutions Provision Combine incentives and autonomy in the T6 Autonomy and Accountability of Public Training Institutions management of public training institutions T7 Introduction and Closure of Public Training Programs Service Delivery Dimension 3 T1 Links between Training Institutions and Industry Integrate industry and expert input into the T2 Industry Role in the Design of Program Curricula Fostering design and delivery of public training programs T3 Industry Role in the Specification of Facility Standards Relevance in G8 T4 Links between Training and Research Institutions Public Training Programs Recruit and support administrators and T5 Recruitment and In-Service Training of Heads of Public Training Institutions instructors for enhancing the market- relevance of public training programs T6 Recruitment and In-Service Training of Instructors of Public Training Institutions Enhancing Expand the availability and use of policy- T1 Administrative Data from Training Providers Evidence-based relevant data for focusing providers' attention G9 T2 Survey and Other Data Accountability on training outcomes, efficiency and for Results innovation T3 Use of Data to Monitor and Improve Program and System Performance 49 Appendix 2: Examples of Benchmarking Rubrics for Topic-Level Data Functional Dimension 1 Level of Development Policy Goal Topic Latent Emerging Established Advanced G1_T1: Advocacy No visible champions advocate Some visible champions advocate Government leaders exercise Both government and non- for WfD to support economic for WfD to support economic sustained advocacy for WfD with government leaders exercise for WfD to development. development on an ad-hoc and support from non-government sustained advocacy for WfD, and Support limited basis. leaders, and collaborate on the rely on routine, institutionalized Economic WfD policy agenda for selected processes to collaborate on an Policy Goal Development industries or economic sectors. economy-wide WfD policy 1: Setting a agenda. Strategic G1_T2: Strategic WfD champions have taken no WfD champions have taken WfD champions have taken WfD champions have taken Direction for Focus and specific action to advance specific action on strategic WfD specific action on strategic WfD specific action on strategic WfD strategic WfD priorities. priorities through a few priorities through a range of priorities through a well- WfD Decisions by the interventions, but no interventions, and integrated range of WfD Champions arrangements exist to monitor implementation progress is interventions, and and review implementation monitored, albeit through ad-hoc implementation progress is progress. reviews. monitored through routine, systematic reviews. Functional Dimension 1 Level of Development Policy Goal Topic Latent Emerging Established Advanced G2_T1: Overall There is limited or no formal Either the government or other Both the government and other The government and other WfD assessment of the country's WfD stakeholders conduct WfD stakeholders conduct stakeholders, as well as Policy Goal Assessment of economic prospects and their occasional assessments of the routine assessments of the independent organizations 2: Fostering a Economic implications for skills. country's economic prospects and country's economic prospects and conduct routine assessments of Prospects and skills implications for a few skills implications for key growth the country's economic prospects Demand- Skills industries, based on limited data. sectors, based on multiple data and economy-wide skills Driven Implications sources. implications, based on comprehensive data. Approach to WfD 50 Functional Dimension 2 Level of Development Policy Goal Topic Latent Emerging Established Advanced G2_T2: Critical The government or WfD The government or WfD The government or WfD The government or WfD stakeholders have not identified stakeholders have identified stakeholders have identified stakeholders have identified Skills Constraints critical skills constraints in priority critical skills constraints in priority critical skills constraints in the key critical skills constraints in the key in Priority economic sectors. economic sectors on the basis of priority economic sectors, based priority economic sectors, based Economic Sectors ad hoc assessments; measures to on formal, well-informed on formal, well-informed address the identified constraints assessments; selected measures assessments; a wide range of are limited to the reform of are being taken to address the measures to address the skills existing courses. skills constrains (e.g., reforming constrains benefit from visible existing courses, dedicated and sustained support from budgets); WfD champions are WfD champions. associated with these measures to some extent. G2_T3: Role of Industry/employers have limited Industry/employers help define Industry/employers help define Industry/employers help define or no role in defining strategic WfD priorities on an ad-hoc basis WfD priorities on a routine basis WfD priorities on a routine basis Policy Goal Employers and WfD priorities. and are making limited and are making some and are making significant 2: Fostering a Industry contributions to address skills contributions in selected areas to contributions in multiple areas implications of major address skills implications of to address skills implications of Demand- policy/investment decisions. major policy/investment decisions. major policy/investment decisions. Driven Approach to G2_T4: Skills- The government provides no The government provides some The government provides a range The government provides a range WfD upgrading incentives for formal and informal incentives for skills upgrading for of incentives for skills upgrading of incentives for skills upgrading sector employers to develop and formal and informal sector for formal and informal sector for formal and informal sector Incentives for upgrade skills. employers. The levy-grant scheme, employers, including a levy-grant employers, including a levy-grant Employers if included, is of limited scheme with broad coverage. scheme with comprehensive coverage. coverage for which an annual report is available. G2_T5: There is no evidence of reviews Incentive programs to encourage Incentive programs to encourage Incentive programs to encourage of incentive programs to skills upgrading by employers are skills upgrading by employers are skills upgrading by employers are Monitoring of the encourage skills upgrading by reviewed for impact, albeit not systematically reviewed for systematically reviewed for Incentive employers. systematically. impact; adjustments are made to impact; adjustments are made to Programs the programs following the the programs following the reviews, with a limited focus on reviews, with an explicit focus on skills and productivity. skills and productivity. 51 Appendix 3: Composite Rubrics for the Policy Actions Functional Dimension 1: Strategic Framework Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Provide Visible champions for WfD are either Some visible champions provide ad- Government leaders exercise Both government and non- Strategic Direction G1: Articulating a absent or take no specific action to hoc advocacy for WfD and have acted sustained advocacy for WfD with government leaders exercise sustained advance strategic WfD priorities. on few interventions to advance occasional, ad-hoc participation from sustained advocacy for WfD, and rely advocacy for strategic WfD priorities; no non-government leaders; their on routine, institutionalized processes arrangements exist to monitor and advocacy focuses on selected industries to collaborate on well-integrated WfD at the top review implementation progress. or economic sectors and manifests interventions to advance a strategic, leadership itself through a range of specific economy-wide WfD policy agenda; interventions; implementation progress implementation progress is monitored level is monitored, albeit through ad-hoc and reviewed through routine, reviews. institutionalized processes. Functional Dimension 1: Strategic Framework Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Establish There is limited or no formal Either the government or other WfD Both the government and other WfD The government and other WfD assessment of the country's economic stakeholders conduct occasional stakeholders conduct routine stakeholders, as well as independent clarity on the G2: Fostering a Demand-Driven prospects and their implications for assessments of the country's economic assessments of the country's economic organizations conduct routine demand for skills; thus critical skills constraints prospects and skills implications for a prospects and skills implications for assessments of the country's economic have not been identified. few industries, based on limited data key growth sectors, based on multiple prospects and economy-wide skills skills and sources; some critical skills constraints data sources; critical skills constraints implications, based on comprehensive Approach to WfD areas of have been identified through ad-hoc have been identified through formal, data sources; critical skills constraints assessments and measures to address well-informed assessments; selected have been identified through formal, critical these constraints are limited (e.g., measures (e.g., reform of existing well-informed assessments; a wide constraint reform of existing training courses). courses, dedicated budgets), supported range of measures, benefitting from to some extent by WfD champions, are visible and sustained support from taken to address the identified skills WfD champions, are taken to address constraints. the identified skills constraints. 52 Functional Dimension 1: Strategic Framework Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Industry/employers have limited or no Industry/employers help define WfD Industry/employers help define WfD Industry/employers help define WfD Engage G2: Fostering a Demand- Driven Approach to WfD role in defining strategic WfD priorities; priorities on an ad-hoc basis and make priorities on a routine basis and make priorities on a routine basis and make employers in the government either provides no limited contributions to address skills some contributions in selected areas to significant contributions in multiple incentives to encourage skills upgrading implications of major policy/investment address the skills implications of major areas to address the skills implications of setting WfD by employers or conducts no reviews of decisions; the government provides some policy/investment decisions; the major policy/investment decisions; the priorities and such incentive programs. incentives for skills upgrading for formal government provides a range of government provides a range of and informal sector employers; if a levy- incentives for skills upgrading for all incentives for skills upgrading for all in enhancing grant scheme exists its coverage is employers; a levy-grant scheme with employers; a levy-grant scheme with skills- limited; incentive programs are not broad coverage of formal sector comprehensive coverage of formal sector systematically reviewed for impact. employers exists; incentive programs are employers exists; incentive programs to upgrading for systematically reviewed and adjusted; encourage skills upgrading are workers an annual report on the levy-grant scheme systematically reviewed for impact on is published with a time lag. skills and productivity and are adjusted accordingly; an annual report on the levy- grant scheme is published in a timely fashion. Functional Dimension 1: Strategic Framework Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Formalize key Government ministries and agencies Government ministries and agencies Government ministries and agencies Government ministries and agencies Coordination for Implementation responsible for WfD have overlapping responsible for WfD have overlapping responsible for WfD have overlapping responsible for WfD have well defined WfD roles for mandates and no mechanisms exist mandates and rely on ad-hoc mandates in a few areas and rely on mandates with hardly any overlap and G3: Strengthening Critical coordinated for coordinating WfD strategies and mechanisms for coordination; non- institutionalized mechanisms for rely on institutionalized mechanisms programs; non-government WfD government WfD stakeholders have no coordination; a few non-government for coordination; key non-government action on stakeholders have no legally-defined legally-defined roles and WfD stakeholders have legally- WfD stakeholders have legally- strategic roles and responsibilities. responsibilities and coordinate with defined roles and responsibilities and defined roles and responsibilities and government through ad-hoc coordinate with government through coordinate with government through priorities mechanisms; strategic WfD measures institutionalized mechanisms; strategic institutionalized mechanisms; are accompanied by an WfD measures are accompanied by an strategic WfD measures are implementation plan and budget with implementation plan and budget with accompanied by an implementation ad-hoc monitoring of progress. routine monitoring of progress. plan and budget with routine monitoring of progress and successful features are mainstreamed into the WfD system. 53 Functional Dimension 2: System Oversight Policy Level of Development Policy Action Goal Latent Emerging Established Advanced G4: Ensuring Efficiency and Equity in Provide stable Government officials determine Budgeting processes for IVET is Budgeting processes for IVET is Budgeting processes for IVET is funding for IVET, CVET and routine but based largely on the routine and based on multiple routine and based on comprehensive funding for ALMPs based on ad-hoc budgeting previous year's budget; recurrent criteria that are occasionally criteria that are routinely reviewed effective processes; there are no government funding for CVET (including OJT in reviewed; recurrent funding for and adjusted; recurrent funding for programs in supported programs for OJT in SMEs) relies on formal processes CVET (including OJT in SMEs) CVET (including OJT in SMEs) SMEs; ALMPs are implemented involving only government officials; relies on formal processes involving relies on formal processes involving initial, through limited arrangements. funding for ALMPs is determined government officials with input government officials with input continuing and through an ad-hoc process involving from key stakeholders and annual from key stakeholders and timely Funding only government officials and reporting with a lag; funding for annual reporting; funding for targeted targets selected population groups ALMPs is determined through a ALMPs is determined through a vocational through a variety of implementation systematic process involving systematic process involving education and arrangements. government officials with key government officials with key stakeholders and targets diverse stakeholders and targets diverse training population groups through a variety population groups through a variety of implementation arrangements; of implementation arrangements; programs are reviewed for impact programs are reviewed for impact with limited follow-up. and adjusted accordingly. Functional Dimension 2: System Oversight Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Monitor and There are no recent formal reviews Recent reviews considered the impact Recent reviews considered the impact Recent reviews considered the impact G4: Ensuring Efficiency and Equity in Funding of the impact of funding on the of funding on training beneficiaries, of funding on training beneficiaries of funding on training beneficiaries enhance equity beneficiaries of training programs. focusing mostly on training-related focusing on both training-related focusing on a full range of training- in funding for outcomes (e.g. enrollment, and labor market outcomes; the related and labor market training completion); the reviews stimulated reviews stimulated dialogue among outcomes; the reviews stimulated dialogue among some WfD WfD stakeholders and some broad-based dialogue among WfD stakeholders. recommendations were stakeholders and key implemented. recommendations were implemented. 54 Functional Dimension 2: System Oversight Policy Policy Action Level of Development Goal Latent Emerging Established Advanced Facilitate The government does not facilitate The government facilitates formal The government facilitates formal The government facilitates formal Efficiency and Equity formal partnerships between training partnerships between individual partnerships between training partnerships between training sustained providers and employers. training providers and employers, providers and employers at the providers and employers at all levels G4: Ensuring partnerships generating some benefits for both institutional as well as at the (institutional, regional, national), in Funding between parties. regional or national levels, generating a broad range of mutual generating a range of mutual benefits for both parties and for the training benefits for both parties. system as a whole. institutions and employers Functional Dimension 2: System Oversight Policy Policy Action Level of Development Goal Latent Emerging Established Advanced Broaden the Competency standards have not been Competency standards exist for a few Competency standards exist for most Competency standards exist for most G5: Assuring Relevant and defined; policy dialogue on occupations; policy dialogue on occupations; policy dialogue on occupations; policy dialogue on scope of competency standards and/or the competency standards and/or the competency standards and/or the competency standards and/or the Reliable Standards competency NQF is limited and occurs on an ad- NQF engages a few stakeholders NQF engages numerous NQF engages all key stakeholders standards as a hoc basis with limited engagement and occurs on an ad-hoc basis; some stakeholders and occurs through and occurs through institutionalized of stakeholders; training providers do training providers offer programs institutionalized processes; the processes; the NQF, if in place, basis for not follow competency-based utilizing competency-based NQF, if in place, covers a few covers most occupations and a wide developing curricula. curricula. occupations and a limited range of range of skill levels; training skill levels; some training providers providers offer programs utilizing qualifications offer programs utilizing competency-based curricula frameworks competency-based curricula. aligned to the agreed standards. 55 Functional Dimension 2: System Oversight Policy Policy Action Level of Development Goal Latent Emerging Established Advanced Establish Skills testing for major occupations is Skills testing for major occupations is Skills testing for major occupations Skills testing for major occupations mainly theory-based; certificates mainly theory-based; competency- assess both theoretical knowledge assess both theoretical knowledge G5: Assuring Relevant and protocols for awarded have little impact on based testing is used to certify and practical skills; competency- and practical skills; competency- assuring the employment and earnings and are qualifications in a few skilled and based testing follows standardized based testing follows robust Reliable Standards credibility of only recognized by public sector semi-skilled occupations; procedures and is widely used in protocols for assuring credibility, employers. certificates awarded are recognized most skilled and semi-skilled including random audits, and is skills testing by public and some private sector occupations; certificates awarded are widely used in most skilled and and employers and have little impact on recognized by both public and semi-skilled occupations; employment and earnings. private sector employers and may certificates awarded are valued by certification impact employment and earnings. both public and private sector employers and consistently improve employment prospects and earnings. Functional Dimension 2: System Oversight Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Develop and No system is in place to establish A designated office in the ministry is An accreditation agency is An accreditation agency is G5: Assuring Relevant and accreditation standards. responsible for defining accreditation responsible for defining accreditation responsible for defining accreditation enforce standards; standards are not standards with stakeholder input; standards through consultation with accreditation consistently publicized or enforced; standards are reviewed on an ad-hoc stakeholders; standards are reviewed Reliable Standards standards for some incentives are offered to seek basis and are publicized or enforced following established protocols and and retain accreditation and only to some extent; training providers are publicized and enforced to a maintaining private providers are required to do are offered incentives and limited large extent; training providers are the quality of so. support to seek and retain offered incentives and support to accreditation and all training seek and retain accreditation and all training providers receiving public funding training providers are required to do provision are required to do so. so. 56 Functional Dimension 2: System Oversight Policy Level of Development Policy Action Goal Latent Emerging Established Advanced G6: Diversifying Pathways for Promote Students in technical and vocational Students in technical and vocational Students in technical and vocational Students in technical and vocational education have limited or no options education can only progress to education can progress to education can progress to educational for further formal skills acquisition vocationally-oriented, non- vocationally-oriented programs, academically or vocationally- progression beyond the secondary level and the university programs; the including at the university level; the oriented programs, including at the Skills Acquisition and government takes no action to government takes limited action to government takes some action to university level; the government improve public perception of TVET. improve public perception of TVET, improve public perception of TVET, takes coherent action on multiple permeability including efforts to diversify learning including efforts to diversify learning fronts to improve public perception through pathways. pathways, which are reviewed for of TVET, including efforts to impact on an ad-hoc basis. diversify learning pathways, which multiple are reviewed for impact on an pathways, routine basis and adjusted including for accordingly. TVET students Functional Dimension 2: System Oversight Policy Level of Development Policy Action Goal Latent Emerging Established Advanced G6: Diversifying Pathways for Facilitate life- Certificates for technical and Some certificates for technical and Most certificates for technical and Most certificates for technical and vocational programs are not vocational programs are recognized vocational programs are recognized vocational programs are recognized long learning recognized in the NQF; in the NQF; few qualifications in the NQF; a large number of in the NQF; a large number of through qualifications certified by non- certified by non-Education ministries qualifications certified by non- qualifications certified by non- Skills Acquisition articulation of Education ministries are not are recognized by formal programs Education ministries are recognized Education ministries are recognized recognized by formal programs under the Ministry of Education; by formal programs under the and granted credits by formal skills under the Ministry of Education; policymakers pay some attention to Ministry of Education albeit without programs under the Ministry of certification recognition of prior learning receives the recognition of prior learning and the granting of credits; policymakers Education; policymakers pay limited attention. provide the public with some pay some attention to the recognition sustained attention to the recognition and information on the subject. of prior learning and provide the of prior learning and provide the recognition of public with some information on the public with comprehensive prior learning subject and there is a formal information on the subject and there association of stakeholders dedicated is a national organization of to adult learning issues. stakeholders dedicated to adult learning issues. 57 Functional Dimension 2: System Oversight Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Provide The government provides practically The government supports a limited The government supports a limited The government supports a G6: Diversifying Pathways no support for further occupational menu of services for further menu of services for further comprehensive menu of services for support for Skills Acquisition and career development, or training occupational and career development, occupational and career development, further occupational and career services for programs for disadvantaged which are available through stand- which are available through an development, including online skills populations. alone local service units or centers; integrated network of centers; resources, which are available training programs targeted to training programs targeted to through an integrated network of acquisition by disadvantaged populations receive disadvantaged populations receive centers; training programs targeted to workers, job- ad-hoc support. systematic support and are reviewed disadvantaged populations receive for impact on an ad-hoc basis. systematic support with multi-year seekers and budgets and are routinely reviewed the and adjusted accordingly. disadvantaged Functional Dimension 3: Service Delivery Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Encourage and Training occurs through state Non-state providers offer training, Non-state providers offer training and A diversity of non-state providers G7: Enabling Diversity and provision only. but few are registered and licensed; some are registered and licensed; the offer training and most are registered regulate non- the government provides limited government provides a range of and licensed; the government Excellence in Training state provision incentives to encourage non-state financial and non-financial incentives provides comprehensive financial of training training provision and limited to encourage non-state training and non-financial incentives to measures are in place to assure its provision and takes systematic encourage non-state training Provision quality; the government conducts measures to assure its quality provision and takes systematic, occasional reviews of its policies on including some measures targeted to robust measures, including regular non-state training provision. lagging institutions; the government reporting and random audits, to conducts routine reviews of its assure its quality; multiple measures policies on non-state training are taken to address quality issues in provision. lagging institutions; the government conducts routine reviews of its policies and adjusts them accordingly. 58 Functional Dimension 3: Service Delivery Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Combine Public training institutions have Public training institutions have some Public training institutions have some Public training institutions have Excellence in Training Provision limited or no autonomy and autonomy; some are governed by a autonomy including over some significant autonomy over incentives and G7: Enabling Diversity and performance targets are not management board, and are allowed aspects of staffing; most are admissions, operations and staffing; autonomy in the specified; the government approves to generate revenues; public training governed by a management board, most are governed by a management management of and closes programs on the basis of institutions are expected to achieve and some are allowed to generate board and generate revenues; public public training ad-hoc processes that are not well- basic targets and are rewarded for revenues; public training institutions training institutions are incentivized informed. doing so; the government uses ad- are expected to and rewarded for to achieve a variety of targets institutions hoc processes informed by some achieving basic targets, with poor through performance-based assessment of implementation performing institutions receiving funding, public recognition and the constraints (e.g. funding, capacity) to support; the government uses closure of poor-performing open and close programs. systematic processes informed by programs; the government uses some assessment of implementation systematic and time-bound constraints (e.g. funding, capacity) as processes informed by assessment of well as formal analysis and implementation constraints (e.g. stakeholder input to open and close funding, capacity) as well as formal programs. analysis and stakeholder input to open and close programs. Functional Dimension 3: Service Delivery Policy Policy Action Level of Development Goal Latent Emerging Established Advanced Integrate Weak or no links exist between Informal links exist between training Formal links exist between some Formal links exist between most training institutions and industry and institutions and industry and research training institutions and industry training institutions and industry G8: Fostering Relevance in Public Training Programs industry and research institutions. institutions; industry provides inputs leading to collaboration in several leading to significant collaboration expert input on an informal basis into the design areas; industry provides inputs in a wide range of areas; industry into the design of program curricula and facility through formal processes into the provides inputs through formal standards for publicly-funded design of program curricula and processes into the design of program and delivery of training programs in some facility standards for publicly-funded curricula and facility standards for public training institutions. training programs in some publicly-funded training programs in institutions; formal links exist most institutions; formal links exist programs between training and research between training and research institutions. institutions leading to significant collaboration. 59 Functional Dimension 3: Service Delivery Policy Level of Development Policy Action Goal Latent Emerging Established Advanced Recruit and Heads and instructors of public Heads and instructors of public Heads and instructors of public Heads of public training institutions training institutions are not recruited training institutions are recruited on training institutions are recruited on are recruited on the basis of support G8: Fostering Relevance in Public Training Programs on the basis of explicit standards; the basis of minimum academic the basis of minimum academic minimum academic qualifications administrators they have few or no opportunities qualifications and they have some, qualifications in addition to in addition to industry or teaching and instructors for in-service training. albeit limited, opportunities for in- industry or teaching experience, experience and receive regular, service training. and receive regular in-service highly diverse in-service training; for enhancing training. instructors are recruited on the basis the market- of minimum academic qualifications in addition to relevance of industry and teaching experience, public training and have opportunities for regular programs in-service training, including periodic industry attachments. Functional Dimension 3: Service Delivery Policy Level of Development Policy Action Goal Latent Emerging Established Advanced There are no specific data collection Training providers collect and Training providers collect and Training providers collect and Accou Expand the based ce- and reporting requirements, but report administrative data and report administrative and other report administrative and other availability training providers maintain their own there are significant gaps in data (e.g., job placement statistics, data (e.g., job placement statistics, 60 and use of databases; the government does not reporting by non-state providers; earnings of graduates) and there are earnings of graduates) and there are conduct or sponsor skills-related some public providers issue annual some gaps in reporting by non-state few gaps in reporting by non-state policy-relevant surveys or impact evaluations and reports and the government providers; most public providers providers; most public providers data for rarely uses data to monitor and occasionally sponsors or conducts issue internal annual reports and the issue publicly available annual focusing improve system performance. skills-related surveys; the government routinely sponsors reports and the government routinely government does not consolidate skills-related surveys; the sponsors or conducts skills-related providers' data in a system-wide database and government consolidates data in a surveys and impact evaluations; the attention on uses mostly administrative data to system-wide database and uses government consolidates data in a monitor and improve system administrative data and information system-wide, up to date database training performance; the government from surveys to monitor and improve and uses administrative data, outcomes, publishes information on graduate system performance; the government information from surveys and efficiency and labor market outcomes for some publishes information on graduate impact evaluations to monitor and training programs. labor market outcomes for numerous improve system performance; the innovation training programs. government publishes information on graduate labor market outcomes for most training programs online. 61 Appendix 4: Composite Rubrics for the Policy Goals Functional Dimension 1: Strategic Framework Policy Level of Development Goal Latent Emerging Established Advanced Visible champions for WfD are either Some visible champions provide ad-hoc Government leaders exercise sustained Both government and non-government G1: Setting a Strategic absent or take no specific action to advocacy for WfD and have acted on few advocacy for WfD with occasional, ad- leaders exercise sustained advocacy for Direction for WfD advance strategic WfD priorities. interventions to advance strategic WfD hoc participation from non-government WfD, and rely on routine, priorities; no arrangements exist to leaders; their advocacy focuses on institutionalized processes to collaborate monitor and review implementation selected industries or economic sectors on well-integrated interventions to progress. and manifests itself through a range of advance a strategic, economy-wide WfD specific interventions; implementation policy agenda; implementation progress progress is monitored, albeit through ad- is monitored and reviewed through hoc reviews. routine, institutionalized processes. Functional Dimension 1: Strategic Framework Policy Level of Development Goal Latent Emerging Established Advanced There is no assessment of the country's Some ad-hoc assessments exist on the Routine assessments based on multiple A rich array of routine and robust G2: Fostering a Demand-Led economic prospects and their country's economic prospects and their data sources exist on the country's assessments by multiple stakeholders implications for skills; industry and implications for skills; some measures are economic prospects and their exists on the country's economic employers have a limited or no role in taken to address critical skills constraints implications for skills; a wide range of prospects and their implications for skills; Approach to WfD defining strategic WfD priorities and (e.g., incentives for skills upgrading by measures with broad coverage are taken the information provides a basis for a receive limited support from the employers); the government makes to address critical skills constraints; the wide range of measures with broad government for skills upgrading. limited efforts to engage employers as government recognizes employers as coverage that address critical skills strategic partners in WfD. strategic partners in WfD, formalizes constraints; the government recognizes their role, and provides support for employers as strategic partners in WfD, skills upgrading through incentive formalizes their role, and provides schemes that are reviewed and adjusted. support for skills upgrading through incentives, including some form of a levy-grant scheme, that are systematically reviewed for impact and adjusted accordingly. 62 Functional Dimension 1: Strategic Framework Policy Level of Development Goal Latent Emerging Established Advanced Industry/employers have a limited or Industry/employers help define WfD Industry/employers help define WfD Industry/employers help define WfD G3: Strengthening Critical Coordination for no role in defining strategic WfD priorities on an ad-hoc basis and make priorities on a routine basis and make priorities on a routine basis and make priorities; the government either limited contributions to address skills some contributions in selected areas to significant contributions in multiple provides no incentives to encourage implications of major policy/investment address the skills implications of major areas to address the skills implications of skills upgrading by employers or decisions; the government provides some policy/investment decisions; the major policy/investment decisions; the conducts no reviews of such incentive incentives for skills upgrading for formal government provides a range of government provides a range of programs. and informal sector employers; if a levy- incentives for skills upgrading for all incentives for skills upgrading for all grant scheme exists its coverage is employers; a levy-grant scheme with employers; a levy-grant scheme with Implementation limited; incentive programs are not broad coverage of formal sector comprehensive coverage of formal systematically reviewed for impact. employers exists; incentive programs are sector employers exists; incentive systematically reviewed and adjusted; programs to encourage skills upgrading an annual report on the levy-grant scheme are systematically reviewed for impact is published with a time lag. on skills and productivity and are adjusted accordingly; an annual report on the levy-grant scheme is published in a timely fashion. 63 Functional Dimension 2: System Oversight Policy Level of Development Goal Latent Emerging Established Advanced The government funds IVET, CVET The government funds IVET, CVET The government funds IVET, CVET The government funds IVET, CVET and ALMPs (but not OJT in SMEs) (including OJT in SMEs) and ALMPs; (including OJT in SMEs) and ALMPs; (including OJT in SMEs) and ALMPs; based on ad-hoc budgeting processes, funding for IVET and CVET follows funding for IVET is routine and based on funding for IVET is routine and based on G4: Ensuring Efficiency and Equity in Funding but takes no action to facilitate formal routine budgeting processes involving multiple criteria, including evidence of comprehensive criteria, including partnerships between training providers only government officials with program effectiveness; recurrent funding evidence of program effectiveness, that and employers; the impact of funding on allocations determined largely by the for CVET relies on formal processes are routinely reviewed and adjusted; the beneficiaries of training programs previous year's budget; funding for with input from key stakeholders and recurrent funding for CVET relies on has not been recently reviewed. ALMPs is decided by government annual reporting with a lag; funding for formal processes with input from key officials on an ad-hoc basis and targets ALMPs is determined through a stakeholders and timely annual select population groups through various systematic process with input from key reporting; funding for ALMPs is channels; the government takes some stakeholders; ALMPs target diverse determined through a systematic process action to facilitate formal partnerships population groups through various with input from key stakeholders; between individual training providers and channels and are reviewed for impact but ALMPs target diverse population groups employers; recent reviews considered the follow-up is limited; the government through various channels and are impact of funding on only training- takes action to facilitate formal reviewed for impact and adjusted related indicators (e.g. enrollment, partnerships between training providers accordingly; the government takes action completion), which stimulated dialogue and employers at multiple levels to facilitate formal partnerships between among some WfD stakeholders. (institutional and systemic); recent training providers and employers at all reviews considered the impact of funding levels (institutional and systemic); recent on both training-related indicators and reviews considered the impact of funding labor market outcomes; the reviews on a full range of training-related stimulated dialogue among WfD indicators and labor market outcomes; the stakeholders and some recommendations reviews stimulated broad-based dialogue were implemented. among WfD stakeholders and key recommendations were implemented. 64 Functional Dimension 2: System Oversight Policy Level of Development Goal Latent Emerging Established Advanced Policy dialogue on competency A few stakeholders engage in ad-hoc Numerous stakeholders engage in policy All key stakeholders engage in policy standards and/or the NQF occurs on an policy dialogue on competency standards dialogue on competency standards and/or dialogue on competency standards and/or ad-hoc basis with limited engagement and/or the NQF; competency standards the NQF through institutionalized the NQF through institutionalized of key stakeholders; competency exist for a few occupations and are used processes; competency standards exist for processes; competency standards exist for standards have not been defined; skills by some training providers in their most occupations and are used by some most occupations and are used by G5: Assuring Relevant and Reliable Standards testing for major occupations is mainly programs; skills testing is competency- training providers in their programs; the training providers in their programs; the theory-based and certificates awarded based for a few occupations but for the NQF, if in place, covers some NQF, if in place, covers most are recognized by public sector most part is mainly theory-based; occupations and a range of skill levels; occupations and a wide range of skill employers only and have little impact certificates are recognized by public and skills testing for most occupations levels; skills testing for most occupations on employment and earnings; no system some private sector employers but have follows standard procedures, is follows standard procedures, is is in place to establish accreditation little impact on employment and competency-based and assesses both competency-based and assesses both standards. earnings; the accreditation of training theoretical knowledge and practical theoretical knowledge and practical providers is supervised by a dedicated skills; certificates are recognized by both skills; robust protocols, including office in the relevant ministry; private public and private sector employers random audits, ensure the credibility of providers are required to be accredited, and may impact employment and certification; certificates are valued by however accreditation standards are not earnings; the accreditation of training most employers and consistently consistently publicized or enforced; providers is supervised by a dedicated improve employment prospects and providers are offered some incentives to agency in the relevant ministry; the earnings; the accreditation of training seek and retain accreditation. agency is responsible for defining providers is supervised by a dedicated accreditation standards with stakeholder agency in the relevant ministry; the input; standards are reviewed on an ad- agency is responsible for defining hoc basis and are publicized or enforced accreditation standards in consultation to some extent; all providers receiving with stakeholders; standards are public funding must be accredited; reviewed following established providers are offered incentives and protocols and are publicized and limited support to seek and retain routinely enforced; all training providers accreditation. are required as well as offered incentives and support to seek and retain accreditation. 65 Functional Dimension 2: System Oversight Policy Level of Development Goal Latent Emerging Established Advanced Students in technical and vocational Students in technical and vocational Students in technical and vocational Students in technical and vocational education have few or no options for education can only progress to education can progress to vocationally- education can progress to academically further formal skills acquisition beyond vocationally-oriented, non-university oriented programs, including at the or vocationally-oriented programs, the secondary level and the government programs; the government takes limited university level; the government takes including at the university level; the takes no action to improve public action to improve public perception of some action to improve public perception government takes coherent action on perception of TVET; certificates for TVET (e.g. diversifying learning of TVET (e.g. diversifying learning multiple fronts to improve public technical and vocational programs are pathways); some certificates for technical pathways and improving program perception of TVET (e.g. diversifying G6: Diversifying Pathways for Skills Acquisition not recognized in the NQF; and vocational programs are recognized quality) and reviews the impact of such learning pathways and improving qualifications certified by non- in the NQF; few qualifications certified efforts on an ad-hoc basis; most program quality and relevance, with the Education ministries are not recognized by non-Education ministries are certificates for technical and vocational support of a media campaign) and by formal programs under the Ministry recognized by formal programs under the programs are recognized in the NQF; a routinely reviews and adjusts such of Education; recognition of prior Ministry of Education; policymakers pay large number of qualifications certified efforts to maximize their impact; most learning receives limited attention; the some attention to the recognition of prior by non-Education ministries are certificates for technical and vocational government provides practically no learning and provide the public with recognized by formal programs under the programs are recognized in the NQF; a support for further occupational and some information on the subject; the Ministry of Education, albeit without the large number of qualifications certified career development, or training government offers limited services for granting of credits; policymakers give by non-Education ministries are programs for disadvantaged further occupational and career some attention to the recognition of prior recognized and granted credits by populations. development through stand-alone local learning and provide the public with formal programs under the Ministry of service centers that are not integrated some information on the subject; a Education; policymakers give sustained into a system; training programs for formal association of stakeholders attention to the recognition of prior disadvantaged populations receive ad-hoc provides dedicated attention to adult learning and provide the public with support. learning issues; the government offers comprehensive information on the limited services for further occupational subject; a national organization of and career development, which are stakeholders provides dedicated attention available through an integrated network to adult learning issues; the government of centers; training programs for offers a comprehensive menu of services disadvantaged populations receive for further occupational and career systematic support and are reviewed for development, including online impact on an ad-hoc basis. resources, which are available through an integrated network of centers; training programs for disadvantaged populations receive systematic support with multi-year budgets and are routinely reviewed for impact and adjusted accordingly. 66 Functional Dimension 3: Service Delivery Policy Level of Development Goal Latent Emerging Established Advanced There is no diversity of training There is some diversity in training There is diversity in training provision; There is broad diversity in training provision as the system is largely provision; non-state providers operate non-state training providers, some provision; non-state training providers, Excellence in Training Provision comprised of public providers with with limited government incentives and registered and licensed, operate within a most registered and licensed, operate G7: Enabling Diversity and limited or no autonomy; training governance over registration, licensing range of government incentives, with comprehensive government provision is not informed by formal and quality assurance; public training is systematic quality assurance measures incentives, systematic quality assurance assessment, stakeholder input or provided by institutions with some and routine reviews of government measures and routine review and performance targets. autonomy and informed by some policies toward non-state training adjustment of government policies assessment of implementation providers; public providers, mostly toward non-state training providers; constraints, stakeholder input and basic governed by management boards, have public providers, mostly governed by targets. some autonomy; training provision is management boards, have significant informed by formal analysis of autonomy; decisions about training implementation constraints, stakeholder provision are time-bound and informed input and basic targets; lagging providers by formal assessment of implementation receive support and exemplary constraints; stakeholder input and use of a institutions are rewarded. variety of measures to incentivize performance include support, rewards and performance-based funding. Functional Dimension 3: Service Delivery Policy Level of Development Goal Latent Emerging Established Advanced There are few or no attempts to foster Relevance of public training is enhanced Relevance of public training is enhanced Relevance of public training is enhanced G8: Fostering Relevance relevance in public training programs through informal links between some through formal links between some through formal links between most in Public Training through encouraging links between training institutions, industry and research training institutions, industry and research training institutions, industry and research training institutions, industry and institutions, including input into the institutions, leading to collaboration in institutions, leading to significant Programs research institutions or through setting design of curricula and facility standards; several areas including but not limited to collaboration in a wide range of areas; standards for the recruitment and heads and instructors are recruited on the the design of curricula and facility heads and instructors are recruited on the training of heads and instructors in basis of minimum academic standards standards; heads and instructors are basis of minimum academic and training institutions. and have limited opportunities for recruited on the basis of minimum professional standards and have professional development. academic and professional standards regular access to diverse opportunities and have regular access to opportunities for professional development, including for professional development. industry attachments for instructors. 67 Functional Dimension 3: Service Delivery Policy Level of Development Goal Latent Emerging Established Advanced There are no specific data collection Training providers collect and report Training providers collect and report Training providers collect and report and reporting requirements, but training administrative data and there are administrative and other data (e.g., job administrative and other data (e.g., job providers maintain their own significant gaps in reporting by non-state placement statistics, earnings of placement statistics, earnings of G9: Enhancing Evidence-based databases; the government does not providers; some public providers issue graduates) and there are some gaps in graduates) and there are few gaps in Accountability for Results conduct or sponsor skills-related annual reports and the government reporting by non-state providers; most reporting by non-state providers; most surveys or impact evaluations and occasionally sponsors or conducts skills- public providers issue internal annual public providers issue publicly available rarely uses data to monitor and improve related surveys; the government does not reports and the government routinely annual reports and the government system performance. consolidate data in a system-wide sponsors skills-related surveys; the routinely sponsors or conducts skills- database and uses mostly administrative government consolidates data in a related surveys and impact evaluations; data to monitor and improve system system-wide database and uses the government consolidates data in a performance; the government publishes administrative data and information system-wide, up to date database and information on graduate labor market from surveys to monitor and improve uses administrative data, information outcomes for some training programs. system performance; the government from surveys and impact evaluations to publishes information on graduate labor monitor and improve system market outcomes for numerous training performance; the government publishes programs. information on graduate labor market outcomes for most training programs online. 68 wol rdban. kog r/d e uao c n/ saber h T eSys tem sApproah c o frBee rEduc ao n Resut ls(A S BER)n ii avecole lcs t daa t onthepoli ce isand i nst uo nsofeducao nsyse tm sao rund t he wol rd n a dbenchmarksthemagains tpracc esaso sciat ed wt ih s tudente larnn ig. A S BERai m stogiveal lpare swih t asa tke i n educaonalresul t—f s rom t sudent, sadm n iit srao trs,e tacher, sand pae rntstopolc iymaker, sbusn iesspeope l, n a dpoi l cale laders —a naccess ibe l,detal e id o , be jcv esnaph sotofhoww el lthe poi li cesofther icounty r'seduao c n sy stema reori entedtowarddel ie vrn ig e larnn igforal lchid lrenandyouth. h T s irp e ot rfou cse s p sei c�c aly lonpol i cis e n itheae raofWok rfoc reDee vlopmen. t h T e�ndi ngsi ,ne trpret ao ns,andconcl uso insexpe rss edherei narethos e ofh te auh tor() sanddonotnee cssar iy lrfl e e ctthe i vewsoftheI nten rao nalBanko f rRec onstr uco nandDevl eopment/TheWo rl dB n a kandt isaffil it aedorgn a z iaons,ort hose ofh teExecuv eDie rco t s rofTheWo rd l Bankorh tegovr enmentsh teyrepree snt. h T eWo r ldBankdoesnotguarane tetheaccua rcyofthedatan icludedn i h tisw ok r.Theboundar ie, scol ors, denominaons,andotheri nfom r aon shownon anymap i n h tisworkdo notm i py lanyjude gm entonthepat rof h T eWo r ldBankconcr eningthee lgal stau ts ofanyte rrt ior y orh te endos rem entorac cepa tnceofsuchboundar ie. s