Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004843 IMPLEMENTATION COMPLETION AND RESULTS REPORT < IBRD – 81180> ON A LOAN IN THE AMOUNT OF EUR12 MILLION (US$15.98 MILLION EQUIVALENT) TO Montenegro FOR THE Higher Education and Research for Innovation and Competitiveness Project December 19, 2019 Education Global Practice Europe And Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective {Jun 11, 2019}) Currency Unit = EUR EUR 0.88277 = US$1 FISCAL YEAR July 1 - June 30 Regional Vice President: Cyril E Muller Country Director: Linda Van Gelder Regional Practice Director: Fadia M. Saadah Practice Manager: Harry Anthony Patrinos Task Team Leader(s): Roberta Malee Bassett ICR Main Contributor: Daria Lavrentieva ABBREVIATIONS AND ACRONYMS CoE Center of Excellence CPF Country Partnership Framework CPS Country Partnership Strategy EC European Commission ECT European Credit Transfer and Accumulation System EHEA European Higher Education Area EMF Environmental Management Framework EMP Environment Management Plan ENQA European Association for Quality Assurance in Higher Education EQAR European Quality Assurance Register for Higher Education ERA European Research Area ESG Standards and Guidelines for Quality Assurance in the European Higher Education Area EU European Union GDP Gross Domestic Product GoM Government of Montenegro HE Higher education HEI Higher Education Institution HERIC Higher Education and Research for Innovation and Competitiveness Project IFAC International Federation for Accountants IFR Interim Unaudited Financial Report ISA International Standards for Auditing M&E Monitoring & Evaluation MoE Ministry of Education of Montenegro MONSTAT Statistical Institute of Montenegro MoS Ministry of Science NQAF National Quality Assurance Framework NQF National Qualifications Framework PMT Project Management Team QA Quality Assurance R&D Research & Development RI Research Institute UCG University of Montenegro TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 11 II. OUTCOME .................................................................................................................... 12 A. RELEVANCE OF PDOs ............................................................................................................ 12 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 13 C. EFFICIENCY ........................................................................................................................... 19 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 21 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 21 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 22 A. KEY FACTORS DURING PREPARATION ................................................................................... 22 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 23 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 25 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 25 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 26 C. BANK PERFORMANCE ........................................................................................................... 27 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 28 V. LESSONS AND RECOMMENDATIONS ............................................................................. 29 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 32 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 43 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 45 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 46 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 51 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 52 COMPONENT 1 - HIGHER EDUCATION FINANCE REFORMS AND IMPLEMENTATION OF QUALITY ASSURANCE NORMS ............................................................................................................ 53 SUBCOMPONENT 1.1 - HIGHER EDUCATION FINANCE REFORMS .......................................... 53 SUBCOMPONENT 1.2 - HIGHER EDUCATION QUALITY ASSURANCE AND RELEVANCE ACTIVITIES 54 National Quality Assurance Framework in Montenegro ............................................................. 54 External Evaluation of HEIs in Montenegro................................................................................ 54 Tracer Study ............................................................................................................................. 56 Plagiarism prevention framework ............................................................................................. 56 Development of National Qualifications Framework ................................................................. 57 SUBCOMPONENT 1.3 - COMPETITIVELY AWARDED GRANTS TO INCENTIVIZE PARTICIPATION AND SUPPORT CAPACITY BUILDING ..................................................................................... 57 COMPONENT 2 - HUMAN CAPITAL DEVELOPMENT THROUGH INTERNATIONALIZATION INITIATIVES ......................................................................................................................... 58 SUBCOMPONENT 2.2 - PROMOTING A SCHOLARSHIP PROGRAM FOR MASTER'S, PHD AND POSTDOCTORAL STUDIES .................................................................................................... 58 Scholarship Program for Master’s, PhD and Postdoctoral Studies .............................................. 58 SUBCOMPONENT 2.3 – TECHNICAL ASSISTANCE TO SUPPORT INTERNATIONALIZATION INITIATIVES ......................................................................................................................... 59 Study programmes in English language ..................................................................................... 59 Improving the foreign credential recognition process ................................................................ 60 Scientific Diaspora .................................................................................................................... 60 COMPONENT 3 – ESTABLISHING A COMPETITIVE RESEARCH ENVIRONMENT ........................ 61 SUBCOMPONENT 3.1 - ESTABLISHMENT OF A PILOT CENTRE OF EXCELLENCE ....................... 61 SUBCOMPONENT 3.2 – SUPPORTING COLLABORATIVE RESEARCH AND DEVELOPMENT SUBPROJECTS ...................................................................................................................... 63 SUBCOMPONENT 3.3 – TECHNICAL ASSISTANCE TO STRENGTHEN THE DESIGN AND IMPLEMENTATION OF THE RESEARCH GRANT PROGRAM .................................................... 65 Technical assistance on intellectual property and commercialization ......................................... 65 Study on research equipment and creation of joint research facility .......................................... 66 Quantitative analysis for the Smart Specialization Strategy ....................................................... 66 Activities to support establishment of Science-Technology Park in Podgorica............................. 67 COMPONENT 4 – PROJECT MANAGEMENT AND MONITORING AND EVALUATION ............... 67 SUBCOMPONENT 4.1 – PROJECT MANAGEMENT ................................................................. 67 SUBCOMPONENT 4.2 – PROJECT MONITORING AND EVALUATION ....................................... 68 HERIC results promotional event............................................................................................... 68 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Higher Education Research for Innovation and P122785 Competitiveness Project Country Financing Instrument Montenegro Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Montenegro HERIC PMU Project Development Objective (PDO) Original PDO The Project development objective is to strengthen the quality and relevance of higher education and research in Montenegro throughreforming the higher education finance and quality assurance systems and by strengthening research and development capabilities. Page 1 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 15,980,000 15,980,000 13,651,329 IBRD-81180 Total 15,980,000 15,980,000 13,651,329 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 15,980,000 15,980,000 13,651,329 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 24-Jan-2012 21-May-2012 24-May-2015 30-Mar-2017 30-Jun-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 30-Jun-2016 9.62 Change in Loan Closing Date(s) 22-Oct-2018 14.22 Change in Loan Closing Date(s) Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 28-May-2012 Satisfactory Satisfactory 0 Page 2 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 02 29-Dec-2012 Satisfactory Satisfactory 2.57 03 23-Jun-2013 Satisfactory Satisfactory 2.67 04 04-Jan-2014 Satisfactory Satisfactory 2.67 05 25-Jun-2014 Satisfactory Satisfactory 3.14 06 27-Jan-2015 Satisfactory Satisfactory 5.84 07 03-Aug-2015 Satisfactory Satisfactory 6.70 08 02-Feb-2016 Satisfactory Moderately Satisfactory 7.33 09 27-Aug-2016 Satisfactory Satisfactory 10.24 10 07-Mar-2017 Satisfactory Satisfactory 11.86 11 14-Sep-2017 Satisfactory Satisfactory 12.60 12 26-Apr-2018 Satisfactory Satisfactory 13.14 13 21-Dec-2018 Satisfactory Satisfactory 14.22 14 27-Jun-2019 Satisfactory Satisfactory 14.22 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Public Administration - Education 50 Tertiary Education 50 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Education 100 Access to Education 25 Science and Technology 25 Teachers 25 Standards, Curriculum and Textbooks 25 Page 3 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) ADM STAFF Role At Approval At ICR Regional Vice President: Philippe H. Le Houerou Cyril E Muller Country Director: Jane Armitage Linda Van Gelder Director: Ana L. Revenga Fadia M. Saadah Practice Manager: Alberto Rodriguez Harry Anthony Patrinos Task Team Leader(s): Roberta Malee Bassett Roberta Malee Bassett ICR Contributing Author: Daria Lavrentieva Page 4 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country context. Montenegro is a small, middle income country. The key sectors of Montenegro’s open economy— and potential growth engines over the longer term—include tourism, services and other knowledge-driven industries. At the time of project preparation, the share of Small and Medium Enterprises (SMEs) had been increasing steadily, but these firms remained focused on the domestic market. There was a clear need for SMEs to become more regionally competitive to strengthen their contribution to the national economy. An important mechanism for increasing competitiveness was improving the efficiency and quality of higher education and strengthening the links between research, innovation and business. 2. Sector context. Although the number of highly educated people in the country was increasing, there was a perceived mismatch between the supply and demand of relevant knowledge and skills. Several national initiatives to investigate and improve the labor market relevance of tertiary education were instituted at this time, most notably, the National Strategy for Employment and Human Resources Development 2012-2015.1 Surveys of employers administered by the Employment Agency of Montenegro indicated an enduring dissatisfaction of employers with the level of knowledge and skills of their highly educated staff.2 This pointed to the necessity for further improvements of higher education in Montenegro to ensure that students are graduating with the knowledge and skills demanded by the labor market. The links between academic research and technology transfer and the private sector, especially to SMEs, were underdeveloped. The establishment of stronger relations between the higher education (HE) and research sector and the private sector had the potential not only to improve productivity, promote innovation, enhance competitiveness, but also to drive economic growth. 3. At appraisal, there were three accredited universities in the country: the largest and only public university— University of Montenegro (Univerzitet Crne Gore – UCG); and two private universities, the Mediterranean University (Univerzitet Mediteran - UM) and the University Donja Gorica (UDG). Collectively, these institutions provided 1 “NationalStrategy for Employment and Human Resources Development for the period 2012-2015, adopted by the Government at the end of 2011, constituted a strategic framework for implementing labor market reforms and it was in line with the EU employment policy including the integrated employment policies guidelines that were most recent at the time. The overall aim of the National Strategy for Employment and Human Resources Development for the period 2012-2015 was to raise the level of employment and quality of human resources in Montenegro. The Strategy sets three major priorities…to address the main challenges at the labor market…: increasing employment and reducing the unemployment rate; …improving knowledge, skills and competencies…; and…promotion of social inclusion and poverty reduction.” From “National Strategy for Employment and Human Resources Development 2016-2020” (p. 6.) Retrieved 19 November 2019 from http://www.mek.gov.me/ResourceManager/FileDownload.aspx?rid=331868&rType=2&file=National%20Strategy%20for%20E mployment%20and%20Human%20Resources%20Development%202016-2020.pdf 2 ANKETA POSLODAVACA ZAPOSLENOST I ZAPOŠLJAVANJE 2011/2012, Retrieved 15 November 2019 from http://www.zzzcg.me/wp-content/uploads/2015/12/PLJEVLJA-2011-2012.pdf Page 5 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Montenegro with a university density that was comparable to most other countries in the region.3 There were also eight non-university private institutions. The UCG, established in 1974, comprised 20 faculties, three research institutions (RI), and one independent study program. It enrolled 21,000 students, or 81 percent of the total student population. The academic infrastructure of the public university needed to be modernized and teaching standards had to be upgraded in line with European Union (EU) norms for all EU universities. The private institutions needed regulatory frameworks, which would enable transparency and confidence in their quality to promote healthy competition with the public university. 4. The Government of Montenegro (GoM) had already been investing in the education system at rates comparable to those of other countries in the region, but without similar outcomes. Among obstacles to effective reforms in the education sector were: (i) lack of easily accessible information on spending, staffing, graduation rates, graduate employment and research output in the higher education sector; (ii) inefficient financing mechanisms of public higher education (driven by costs versus strategic objectives); and (iii) internal fragmentation of the public university, in which faculties acted as quasi-autonomous entities. While gross tertiary education enrollment at appraisal was relatively high, at 52 percent (World Economic Forum/Global Competitiveness Report, 2010-11), the higher education system suffered from structural issues similar to those of many other formerly socialist countries. This resulted in comparatively high drop-out rates and deficiencies in education quality and management. The funding model was based on academic and administrative staff costs, where the faculties received budgets based on staff salaries. Moreover, academic staff were allowed to supplement their income through enrollments of fee-paying students, who were admitted without government support. As a result, the incentive structure provided by the funding norms promoted over-enrollment, a focus on teaching over research (for which supplemental funding was almost non-existent and which must be done in addition to managing full teaching loads, as opposed to being in balance with teaching), and minimal concerns about outcomes and quality. 5. Comparatively low spending on research and development (R&D), particularly by the business sector, was constraining Montenegro’s innovative capacity. According to available data, R&D expenditures per annum were less than 0.16 percent of GDP, compared to the EU254 average of 1.86 percent. It fell far short of the 3 percent target of the Lisbon Agenda/Europe 2020 Strategy. Low levels of business R&D were a result of the decline or closure of industrial firms that used to have in-house R&D units; after closure, firms relied on technology acquisition from abroad. At the same time, new knowledge-intensive industries had yet to emerge in significant numbers. Consequently, the share of high technology exports in Montenegro’s export basket remained low. Research capacity was limited: in 2008, Montenegro had 766 full-time researchers, about 0.3 percent of the total workforce. 6. Several factors, common to other Western Balkans countries, were inhibiting R&D in Montenegro, including: a fragmented policy framework focused too heavily on basic, isolated R&D; poor linkages between higher education institutions (HEIs), R&D centers, and private businesses at national and regional levels; insufficient availability of scientists and engineers; dated or inadequate laboratory equipment for the needs of high quality scientific and applied R&D; and, limited technology commercialization and patenting activities. 7. Strengthening the human capital and research infrastructure was a pre-requisite for effective participation in the European Commission’s (EC’s) framework programs and for integration into the European Research Area to advance the GoM’s EU accession agenda. It was imperative for Montenegro to be aligned with the Bologna Process, which, 3 The university density per 100,000 inhabitants in Montenegro is 2.1, while it is 3.0 in Serbia, 1.8 in Bulgaria, 2.6 in Romania, and 1.1 in North Macedonia, for instance. 4 EU members states prior to the 2007 expansion Page 6 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) steered by European Higher Education Ministers, sought to bring more coherence to higher education systems across Europe. The Bologna Process established the European Higher Education Area (EHEA) to facilitate student and staff mobility to make higher education more inclusive and accessible, attractive, and competitive. Quality assurance was key in developing mutual trust as well as increasing mobility and fair recognition of qualifications. The European Association for Quality Assurance in Higher Education (ENQA) was set up to drive the development of quality assurance across all the Bologna Process signatory countries. ENQA members are EHEA quality assurance organizations. The European Quality Assurance Register for Higher Education (EQAR), established by ENQA, is a public register of quality assurance agencies that substantially comply with the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). Most EU HEIs and individual study programs are subject to regular external reviews by a QA agency. To better integrate with EHEA and achieve the most benefits from it, the long-term goal of the GoM was to become a member of ENQA and EQAR. 8. To transform higher education and research into the mainstream economic drivers and to meet the EU Acquis standards, the GoM placed these sectors at the center of the policy agenda. The Strategy for the Development of Financing of Higher Educations (2011) and the Law on Scientific Research Activity (2010) formed the legislative basis for ambitious reforms. In order to provide targeted reforms and ensure strategic leadership in these areas, the GoM split the Ministry of Education and Science into two separate ministries. Each was charged with the implementation of a comprehensive reform strategy aimed at: improving quality, promoting better alignment between higher education and the labor market, fostering closer collaboration between academia and the private sector and applied research and innovation, and furthering integration in the EHEA and the European Research Area (ERA). Following the norms of the EHEA, faculty, student, and scholar internationalization and mobility was set as a cross-cutting goal to ensure better competitiveness and integration with the EU, including comparability in quality assurance standards. 9. The Montenegro Higher Education and Research for Innovation and Competitiveness (HERIC) project emerged as a principal instrument for promoting large-scale improvements in Montenegro’s higher education and research sectors. It sought to harness the potential of Montenegro’s human capital through relevant investments in these sectors and to help the GoM implement the recently adopted legislation. The project supported Priority 1 of the Country Partnership Strategy (CPS) for FY11-14 – Strengthen institutions and competitiveness in line with EU accession requirements. Theory of Change (Results Chain) 10. According to the project’s Theory of Change, improving higher education finance and quality assurance systems would lead to better quality and relevance of higher education while strengthening research and development capabilities would result in better quality and relevance of research. Relevance was defined as the utility of the outputs for stakeholders (HEIs staff and graduates, RIs, MoE, MoS, employers in public and private sectors) and local and international contexts in which the stakeholders operate. The cumulative impact of the project in both higher education and research sectors was envisioned to support the Government’s long-term strategy of nurturing more productive human capital, development of a competitive and innovative economy, and a stronger integration with EU, ultimately leading to the EU accession. The project was designed to support two distinct sectors and their lead institutions, MoE and MoS. Both ministries pursued complementary and yet distinct areas of reforms. Hence, most of the project activities were designed to separately support each of the sectors. Interventions around students’ and researchers’ mobility were envisioned as cross-sectoral. The project treated quality and relevance as mutually reinforcing and dependent concepts and designed the interventions targeting both at the same time. The graphical Page 7 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) depiction of the Theory of Change (Figure 1) was developed based on the Project Appraisal Document (PAD) and interviews with the Bank project team and clients. 11. Interventions in higher education aimed at improving the systems in place to support effective delivery and outcomes from higher education, including reforming the processes for financing the sector, improving quality assurance norms, and designing and operationalizing a National Quality Assurance Framework (NQAF), in line with EHEA norms. Improved quality was designed to be measured through the ability of HEIs to: a) systematically self-evaluate; b) be (re)accredited following external evaluations, which would follow EHEA norms; and c) be gradually advancing in compliance with ESG standards. Relevance was envisioned to be assessed through: a) the application of ESG standards which embed the concept of relevance by design; and b) a perception of higher education programs and degrees among employers. The exact methodology for perception measuring was to be developed during project implementation. 12. Compared to a more advanced higher education sector with its mature institutions, the research and innovation agenda was nascent, including a very young MoS, created as a separate institution at project inception. Therefore, activities supporting the research sector were dedicated to piloting various concepts forming a competitive and innovative research environment, which would be closely linked to the private sector and international partners. R&D partnerships and the CoE pilot aimed at supporting closer linkages between higher education and research. Quality and relevance of research was to be measured through the ability to create such partnerships between Montenegrin academia and the private sector/other partners, which would lead to major scientific and innovation accomplishments or significantly advance it, and which would pilot the process of technology transfer. The project intended to rely on external evaluations to assess the accomplishments of the R&D/CoE interventions. Figure 1: Theory of Change Page 8 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Project Development Objectives (PDOs) 13. The HERIC Project development objective is to strengthen the quality and relevance of higher education and research in Montenegro through reforming the higher education finance and quality assurance systems and by strengthening research and development capabilities. Key Expected Outcomes and Outcome Indicators 14. The achievement of the PDO was measured by the following indicators: • Higher education has better quality and is more relevant as measured by: • National Quality Assurance Framework (NQAF), according to Bologna-defined EU norms, is operational5, measured by: ▪ 2 cycles of external evaluations of 3 universities conducted (baseline, follow-up) ▪ Quality assurance and accreditation measures applied nationally to all HEIs ▪ 3 universities (re)accredited • New HE funding model implemented • 20% increase in positive perception of HE programs/ degrees among employers • Research has better quality and is more relevant as measured by • One pilot CoE established • 10 public/ private or international R&D partnerships or start-ups with RIs established Components 15. Component 1 – Higher Education Finance Reforms and Implementation of Quality Assurance Norms (estimated US$3.73 million and actual US$1.73): was designed to support implementation of key higher education finance and quality reforms, identified in the Strategy for the Development and Financing of Higher Education. It consisted of three sub-components: • Higher Education Finance Reforms: seeking to adapt to relevant and sound global practices, including cost-per- student financing, performance management agreements, and innovation funding, to promote flexibility and innovation at all levels of higher education—universities, institutes, faculties, and academic programs. • Higher Education Quality Assurance and Relevance Activities: aimed at a) improving capacity of the Council of Higher Education, the public, autonomous quality assurance agency for Montenegro, and strengthen QA efforts of the Higher Education Department of the MoE and other relevant stakeholders; b) financing efforts to achieve full compliance with the norms and standards of EQAR and to become a full member of ENQA, including the payment of membership fees; c) conducting external evaluations of all Higher Education Institutions (HEIs), including baseline and follow-up studies, and the external evaluation of Montenegro’s three universities and all academic institutions; and d) financing one tracer study) to help HEIs and MoE better collect, analyze, and disseminate information on Montenegrin graduates’ employment outcome. 5The original composite outcome indicator - Complete external examination of the three main universities, utilizing new quality assurance and accreditation measures developed in accordance with Bologna-defined EU norms and practices - was broken down into three sub-indicators, all of which demonstrate the application of a National Quality Assurance Framework Page 9 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) • Competitively Awarded Grants to Incentivize Participation and Support Capacity Building: aimed at providing technical assistance to prepare an institutional grants scheme, designed to incentivize participation in the reforms and support capacity building. The sub-component financed grants to HEIs/faculties that have improvement plans approved by an Evaluation Committee of experts. 16. Component 2 – Human Capital Development through Internationalization Initiatives (estimated US$2.81 million and actual US$3.30): was designed to maximize absorption of knowledge and technical training provided around the world for students and academic staff through foreign study and research in areas of national importance. It consisted of three sub-components: • Improving the Support for Students and Scholars mobility: financing a) the establishment of an office within the MoE to coordinate bilateral and multilateral programs for students and academic staff; b) minor improvement of existing student hostels. • Promoting a Scholarship Program for Master’s, PhD, and Postdoctoral Studies: aimed at the design of the targeted scholarship scheme as well as financing scholarships to master’s, doctoral, and postdoctoral studies. • Technical and Analytical Activities to Support Internationalization Initiatives: financing a) exploring the feasibility of operating academic programs in English and, based on the results, design and implementation of such programs; b) strengthening the relationship with the Montenegrin Diaspora to promote international opportunities for scholars and students. 17. Component 3 – Establishing a Competitive Research Environment (estimated US$7.94 million and actual US$6.47): designed to help to upgrade Montenegro's science and technology infrastructure in areas where the scientific and economic potential are highest, thereby enhancing the quality and relevance of R&D outputs and increasing cooperation with private sector and foreign research institutions. It consisted of three sub-components: • Establishment of a Pilot Center of Excellence (CoE): supporting the design and establishment of one pilot CoE and fund its partnerships with private sector and international RIs. • Supporting Collaborative Research and Development (R&D) grants: aimed at establishing of a competitive grants program to fund larger, more impactful R&D activities which could lead to international collaboration and the generation of commercial innovations. The objective was also to fill a critical gap in the funding space for research between the existing National Projects and the planned CoEs. The grants were only awarded to teams comprised of researchers working in Montenegro universities, RIs, or firms implementing proposals together with an international partner. • Technical and Analytical Activities to Strengthen the Design and Implementation of the Program: aimed at capacity building within the MoS to administer the new CoE and grant programs. 18. Component 4 – Project Management and Monitoring and Evaluation (estimated US$1.26 million and actual US$1.19): was designed to build capacity within MoE and MoS to manage the day-today implementation of the HERIC project, as well as to monitor and evaluate its impact. It supported the establishment of a Project Management Team (PMT) and included two labor market studies (one benchmark and one results study) to support the quality and relevance considerations under Component 1. Page 10 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 19. The project PDOs or outcome targets were not revised during the life of the project. Revised PDO Indicators 20. The project PDOs Indicators were not revised during the life of the project. Revised Components 21. The project components were not revised during the life of the project. Other Changes 22. The project’s closing date was extended twice for a cumulative addition of 27 months (until June 2019) to allow for the completion of all project activities, including: 1) implementation the National Excellence Scholarship Program, linked to the academic cycle (two-year duration of many overseas master's degree and certificate programs), and its final assessment; 2) establishment of the CoE, completion of its subprojects (experiencing procurement delays), evaluation of the pilot experience, and development of a framework for future centers. The project extension was also used to ensure its sustainability, namely: a) build capacity of the Agency for Control and Quality Assurance of Higher Education; and b) due to savings incurred during the implementation, fund additional tasks - primarily the design for a Science Technology Park in Podgorica. Rationale for Changes and Their Implication on the Original Theory of Change 23. The extension of the project had beneficial effects on achieving the PDO, and additional activities contributed to enhancement of the science and research environment in Montenegro. The PDO would not have been achieved without the extension, because the interventions in components 2 and 3 would not have been implemented. Therefore, the PDO part on strengthening quality and relevance of research would not have been achieved. The additional activities helped build stronger capacity of project’s stakeholders and contributed towards enhancing the sustainability of results. Page 11 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 24. Relevance of the PDO is High. The project was developed within the overarching framework of the GoM’s agenda to integrate with the EU. It followed the EU acquis requirements for different sectors and the “Stabilization and Association process”, which includes additional conditions for the Western Balkan countries. The project supported the GoM’s efforts to meet the EU acquis requirements for three chapters: Chapter 20: Enterprise and industrial policy, Chapter 25: Science and Research, Chapter 26: Education and culture. The EU accession strategy as well as making progress along the EU acquis chapters, remained highly relevant during the project implementation. 25. At preparation, the project provided technical support in development and implementation of the national strategies for higher education and research, prepared with the Bank’s assistance. The strategies focused on enhancing human capital development and research and innovation infrastructure as well as reducing the mismatch between education outcomes and labor market demand to ultimately drive economic expansions and to make the country more competitive vis-à-vis its EU peers, one of the key objectives of the EU acquis requirements. The project provided implementation support to the six out of eight strategic goals, defined in the GoM’s Strategy for Development and Financing of Higher Education, adopted in 2011 while the project was being prepared: 1. Ensuring and improving quality of higher education; 2. Linking higher education and the labor market demand and raising entrepreneurial, innovative character education; 3. Analysis of the results of applying the principles of the Bologna Declaration in Montenegro; 6. Linking higher education with research; 7. Internationalization of higher education; 8. Adequate and appropriate funding of higher education. The project also supported the implementation of the GoM’s strategy for strengthening capacity for R&D, anchored in the 2010 Law on Scientific Research Activity. The project focused on building capacity of the recently created MoS to design and administer new ambitious programs, nurturing the country’s competitive and innovative research environments. 26. During implementation, the project continued to be fully aligned with national development strategies and legislation, which were created as a result of project interventions: Strategy for Development of Higher Education 2016-2020, Strategy for Innovation Activity 2016-2020 with the Action Plan, Law on Innovation Activity adopted in 2016, Strategy for Scientific Research Activity 2017-2021, Smart Specialization Strategy 2019-2024, Law on Higher Education adopted in June 2017, Law on Academic Integrity adopted in March 2019. 27. At appraisal, the project supported Priority 1 of the Country Partnership Strategy (CPS) for FY11-FY15 – Strengthen institutions and competitiveness in line with EU accession requirements. The project remained relevant for the subsequent Country Partnership Framework (CPF) for FY16 – FY20. It supported Focus Area 2 – Expand Access to Economic Opportunities6 and directly contributed to CPF objective 2A: Enhanced quality and relevance of higher education and research 6 as per revised Results Matrix of the August 20, 2019 Performance and Learning Review of the CPF FY16-20 Page 12 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 28. Achievement of the PDO is assessed against two overarching outcomes: a) Higher education has better quality and is more relevant and b) Research has better quality and is more relevant. • Higher education has better quality and is more relevant National Quality Assurance Framework (NQAF), according to Bologna-defined EU norms, is operational (achieved) 29. A new NQAF was established and operationalized, following Bologna-defined EU norms and practices. • Two cycles of external evaluations of all HEIs, including three main universities, were conducted to help achieve full compliance with the norms and standards of ENQA and EQAR (achieved). Recommendations from the initial 2013 evaluations formed the basis of detailed action plans for individual HEIs and MoE for development and application of quality assurance and accreditation measures. To incentivize the implementation of the recommendations, the project funded a successful implementation of 16 competitive grant proposals for institutional improvement of HEIs, ranging from curriculum improvement and strategy development to modernization of IT and laboratory equipment. The follow-up 2018 evaluations showed a significant progress in the implementation of the 2013 recommendations, leading to major improvements in higher education quality. Key results included: the adoption of new legislation and institutional set-up, and the implementation of new models of studies and funding. As accounted by all the interviewed stakeholders, as a biggest outcome, the assessment cycles helped build capacity within MoE and HEIs for systematic self-evaluation and evidence-based decision-making. Assessment of the educational quality in HEIs is now done internally and externally. Internal quality assessment is performed by a HEI itself. External assessment and subsequent (re)accreditation based on its results is carried out by the new national quality assurance agency. MoE regulates methodology for both internal and external evaluation. The National Qualifications Framework, in line with the European Qualification Framework, was originally envisioned to be part of the project but was eventually implemented, utilizing EU funds. However, the concepts and the implementation support of this activity were complementary to the structure and outcome goals of HERIC. • Quality assurance and accreditation measures applied nationally to all HEIs (achieved). Country systems were strengthened to systematically administer and apply quality assurance and accreditation measures to all HEIs, in accordance with EU standards and practices. The implementation of the 2013 recommendations was enforced by the new 2017 Law on Higher Education (effective from academic year 2017/18) and contributed to the development of the Strategy for Development of Higher Education 2016-2020. To strengthen responsibility for ethical behavior at HEIs, the project also helped develop a new framework for plagiarism prevention, which was later prescribed by the 2019 Law on Academic Integrity. The project provided technical assistance to the Council of Higher Education, MoE, and HEIs to meet the requirements for membership in ENQA and EQAR and to be in full compliance with ESG standards and guidelines. To ensure the transparent application of the quality assurance and accreditation measures, the project helped the Council produce relevant procedures, guidelines and templates. In November 2017, the GoM established a new independent Agency for the Control and Quality Assurance of Higher Education (QA Agency), advancing a longer-term goal of establishing an internationally recognized Montenegrin quality assurance body. The QA Agency took over quality assurance responsibilities from Page 13 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) the Council and was charged with conducting (re)accreditation of HEIs based on external evaluations. The project helped build administrative and operational capacity within the QA Agency. At the project closure, the organization was successfully completing another round of accreditation of HEIs, in accordance with the new 2017 Law, and was confirmed in July 2019 as an affiliate member of ENQA, a first step towards the full membership. The 2018 recommendations indicate that governance of HEIs remains decentralized and driven by individual faculties which makes it more challenging to roll out strategic decisions efficiently and effectively and to advance farther in quality assurance. Nevertheless, the 2017 Law clearly allows for more of such institutional reforms to happen in the future; and the implementation of the new funding is an example that governance efforts are improving the progress towards more centralized decision-making. • All HEIs, including four main universities, were (re)accredited in line with EU standards, which are based on the assessment of quality (achieved). The project supported two cycles of accreditation of HEIs, confirmed by independent external evaluations conducted by an EQAR/ENQA member, and helped the country establish a systematic process of (re)accreditation, enforced by the new legal and institutional frameworks. In 2013, all HEIs were reaccredited, following Bologna-defined EU standards. In 2017, to meet the requirements of the new Law on Higher Education, all study programs had to be newly accredited to comply with a new study model. The model was introduced to all HEIs, following the 2013 evaluation recommendations, and was closer aligned with the Bologna Process-defined three-cycle higher education system, dominant among EU HEIs. It required mandatory 25 percent practical education in all HEIs to make students’ skills more relevant to the labor market demand. By the project closure, the newly established QA Agency completed the reaccreditation of the UCG (160 programs, 76 percent of the country’s programs) and was on target to complete the remaining 48 study programs before the end of 2019 (27 completed, 21 are at final stages of the assessment). As a result of the 2013 evaluation recommendations, the number of HEIs and academic programs have been reduced (from 13 to 9 HEIs; from 300+ to 208 study programs), which has also helped with a more effective and efficient application of quality and accreditation measures nationally. New HE funding model implemented (achieved) 30. A new performance-based financial contracting model for public HEIs to improve the quality of public funding was prescribed by the 2017 Law on Higher Education and was adopted by the UCG in December 2018. The goal was to ensure efficient and effective public allocations in line with national priorities. The model introduced allocations towards well-defined in advance strategic plans, which could be traced during the implementation via a more detailed breakdown-cost vs. the lump sum funding based on staff salaries in the past. It also prescribed a cap on student enrolment to improve the student-teacher ratio and enforced free studies to promote the quality of learning outcomes vs fee-based student enrolment. To offset the loss of revenues mainly from fee-based tuitions, the GoM increased the education budget by EUR2m, or more than 11 percent (EUR19.4m in total). To motivate better performance and reduce the drop-out rates, the model also introduced a required minimum of ECTSs (European Credit Transfer and Accumulation System) students have to comply with annually to retain a budget-funded status. Through mobility and internationalization programs, which the project helped streamline and scaled up, MoE also intended to incentivize individual faculties and study programs, especially in science and research, to tap into European grants. The risk of this approach is in sustainability of efforts which depends on stable funding: the state funding remains low and dependent on political priorities; it is too early to assess the capacity of HEIs to systematically received EU and other grants. Despite of the slow start due to intense deliberation within the GoM on economic implications of changes, the model was successfully rolled out before the end of the project. The GoM ’s comprehensive efforts to design and implement it are to be commended. Page 14 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 20% increase in positive perception of HE programs/ degrees among employers (not achieved) 31. Accurately measuring relevance of higher education proved to be difficult. The project PDO indicator did not yield conclusive results. However, the 15 percent decrease in the numbers of unemployed graduates with higher education degree between 2016-2018 may be perceived a positive trend which could indicate the relevance of skills. The project helped build capacity and provided tools to MoE and HEIs for setting up monitoring and evaluation mechanisms on transition and skills mismatch issues. Although the index of relevance of higher education among employers,7 introduced in perception surveys, was not particularly useful, the exercise yielded helpful recommendations, including the need for: systematic engagement of employers in curricula development; in cooperation between HEIs and employers, provision of opportunities for practical training; and plans for university enrollment based on the labor market demand. The new funding model and the mandatory 25 percent practical teaching in all study programs at UCG were the right steps towards addressing these recommendations. The project also funded a comprehensive tracer study for students who graduated in 2009 and 2013. The study focused on their experience after graduation and provided a methodology for data collection and its application to create relevant policies and interventions. MoE intends to repeat the study as well as to apply its methodology in assessing the relevance of vocational and secondary educations. To fully meet the labor market demand, Montenegrin HEIs and MoE need to systematically monitor and evaluate graduate employment and employer’s feedback. It should help with the evidence-based decision-making on how to best continuously calibrate the type and quality of HEIs’ offerings and learning outcomes. • Research has better quality and is more relevant One pilot CoE established (achieved) 32. The first CoE in the country was established to foster research excellence and cooperation between academia and the private sector. It was a core component of the GoM’s science strategy, articulated in the Law on Scientific Research Activity. Based on the competitive selection process, the new CoE was set up at the UCG and implemented by the Faculty of Electrical Engineering. The new BIO-ICT Center focused on boosting the application of the latest ICT innovations in sustainable agriculture, forestry, water, health, and land management. It was the first biggest multi- disciplinary research initiative funded by the GoM. It involved 90 researchers, including 31 young scientists and students, and was an example of how to build stronger linkages between higher education and science. The Center helped elevate the research agenda in the country and achieved many results: 4 new laboratories established; scientific equipment upgraded for 10 existing laboratories; 51 scientific papers published (out of which 31 on the SCI list) and 13 theses defended based on the research advanced by the Center; 5 patents issued at the national level and one international patent in the process of being approved; 7 new innovative products and services developed, 5 of which applied in the private and public sectors. 33. The CoE concept was ambitious and challenging to implement in a sustainable way; yet it helped the GoM to develop a new model of R&D and innovation in the country. The CoE project experienced several challenges: a large grant with limited capacity to be absorbed in a timely and efficient way; limited understanding of market capabilities; lack of innovation, institutional and legislative frameworks; limited scientific infrastructure in the country; and high 7The 2013 baseline and 2018 follow-up surveys on the perception of relevance (or utility) of higher education by employers in Montenegro did not produce statistically different results. With a highly favorable baseline (the relevance index of 86 out of 100), both surveys concluded that there was not a big gap between the needs of employers and educational outcomes and most employers found university education very relevant. Page 15 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) project transaction and administrative costs. Unclear legal and institutional arrangements within UCG during most of the BIO-ICT center lifetime were one of the biggest impediments of the CoE sustainability. However, this was resolved on May 10, 2018, when the GoM established a special organizational unit within UCG, Institute – Center of Excellence for Research and Innovation, with a broader mandate to include all UCG research disciplines and groups. It took over the resources and research concept of the BIO-ICT center. As part of the new Institute, a new technology transfer office is in the process of being established to help with commercialization of knowledge created by Montenegrin scientists. The pilot challenges gave valuable lessons to the GoM, which were addressed during the implementation of the next generation of CoEs. In September 2018, MoS announced a call for establishing new CoEs, with an objective to launch several centers, which are envisioned to be: more focused (as opposed to the highly exploratory BIO-ICT center); targeting strategic partnerships; and supported by smaller grants. A CoE framework established by HERIC as well as lessons learned from other project’s competitive grants programs were applied: a comprehensive methodology and principles for selecting suitable proposals, a funding model based on business-co- financing, and robust monitoring and evaluation processes and procedures. 10 public/ private or international R&D partnerships or start-ups with RIs established (achieved) 34. Twenty public/private or international R&D partnership with domestic RIs were established, surpassing the original target of 10. Through HERIC support, MoS designed a new Program for Collaborative R&D Grants (CRDS). Like the CoE establishment process, a well-thought through competitive grants scheme was developed and applied, including the principle of partner in kind contributions/ co-financing. CRDS financed proposals in the national priority areas: Energy, ICT, Medicine and Health, Sustainable Development and Tourism, Agriculture and Food, and New Materials, Products, & Services. During the implementation, larger than originally designed sub-projects with more partnership potential, were awarded (8 versus 11), which did not negatively affect the original project goals. CRDS engaged 146 researchers, including 61 young scientists and students, and contributed to stronger linkages between higher education and science. An intendent external evaluation of CRDS notes that the program created a significant impact ranging from development of patented novel methods for mosquito control to creation of a start-up company with a new technology for monitoring of cardiac patients. CRDS results include: 7 new laboratories established; scientific equipment upgraded for 4 existing laboratories; 317 scientific papers published (out of which 89 on the SCI list); 11,766 citations received; 15 theses defended based on the research advanced by individual sub-projects; 2 patents issued at the national level and one patent internationally; 15 new innovative products and services developed, most of which applied in the private and public sectors. Following the CRDS experience, MoS established a new Innovation Projects Grant Program (2019-2024), already awarded grants to 10 competitively selected proposals, and, in July 2019, launched a second call for proposals. Applying lessons learned from CRDS, partners continue to co-finance (in the first round, about 60 percent came from MoS, while the rest was from partner). 35. HERIC helped strengthen national upstream R&D systems which are now focused on systematic and continuous improvement of quality and relevance of research and serve as main innovation drivers. Through CoE and CRDS pilots, the project helped create and operationalize new funding tools for competitive and innovative research, upgraded science and technology infrastructure and institutions, and increased collaboration between science and private sectors. Through a big infusion of funds, HERIC revitalized the engines of the research sector and facilitated many scientific outputs, including doubling the number of country’s publications from project start until closing. In depth analysis of two CRDS projects leading in the number of SCI publications (CS-ICT and LOVCEN) between 2011- 2018 reveals their significant impact on the total number of publications in Montenegro for that period and attests to the overall high quality of science. CS-ICT contributed 24.3 percent of total number of publications in computer Page 16 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) science (signaling), while LOVCEN contributed 19.5 percent to total number of publications in agriculture (insect science). Some of the projects were very successful in patenting and greatly contributed to the process of technology transfer in Montenegro. MoS streamlined the project’s accomplishments and lessons learned into longer-term sustainable programs. Its budget was increased by 60 percent, in 2018; and national programs have been growing in funding and number, most of them based on the HERIC principle of business co-financing. The project helped the GoM with a strategic direction for R&D and innovation, by contributing to: 2016-20 Strategy for Innovation, 2016 Law on Innovation Activity (first time in the country), Strategy for Scientific Research Activity 2017-21, and Smart Specialization Strategy 2019-24 (S3.me). S3 is focused on the new model of economic development based on targeted support to scientific research activities and innovations. It creates a synergy between the European policies and sources of financing, complementing national and regional programs and private investments. With the project support, Montenegro was the first (and the only) country in the Western Balkans to map national R&D infrastructure to avoid procuring the same capital equipment and optimize its use through equipment sharing and the first non- EU28 country to adopt a national Smart Specialisation Strategy (S3). MoS should continue funding the updates of the national research equipment database. HERIC also supported the design and establishment of the Science and Technology Park of Montenegro, one of the priority infrastructure initiatives by the GoM, launched in 2019 to support science and innovation. 36. HERIC helped promote research agenda in HEIs and build stronger linkages between higher education and research; the GoM now needs to sustain and deepen these efforts. Although UCG remains the only HEI in the country with research activities of any significance, the three other universities intend to build research capacity and doctoral education. The 2018 evaluation of HEIs reported a significant growth in research activities with more publications and international cooperation. HERIC helped define connection mechanisms between higher education and research, articulated in the UCG Rulebook on Research and Innovation Processes: professors now have to involve at least two students in all research activities and explain the choice of topic for Master’s and PhD theses. UCG is working on a strategy with a set of actions for MoE and MoS to position the organization not only as a teaching but also as a lead scientific institution. The Strategy for Scientific Research Activity (2017-2021) envisions increased employment of researchers at universities, including full-time. The GoM should sustain these efforts and deepen the connection between research and higher education by: creating a better balance between teaching (currently dominant) and research; acknowledging research as a positive impact on the quality of teaching; providing a separate budget line for research staff, 60 percent of whom are employed in higher education and are funded by MoE; better recognizing research activity as part of staff promotion; and creating more synergies between MoE and MoS policies and actions. The GoM should also encourage UCG management to provide stronger support to research groups; it was perceived to be suboptimal by CRDS grant recipients compared to that one provided private HEIs. 37. The project initiatives across higher education and research to improve mobility and connectivity of researchers, students, and faculty helped advance both PDO outcomes. Due to the country’s small size, international linkages are necessary to leverage international knowledge and expertise, as well as to expand access to a broader network of business partners, customers, and investors. Hence, the project primarily focused on advancing outgoing mobility. • A joint MoE/ MoS National Excellence Scholarship Program (NESP) for MSc, PhD, and Postdoctoral studies was established following best practices in the EU and funded 132 students to study abroad, mainly in European HEIs. An independent external evaluation of NESP (June 2019) concludes that the original NESP objective of gaining new knowledge through academic development and transferring it back into the national science and innovation environment was fully achieved. NESP resulted in more than 150 papers published/ presented; representation of Montenegro in 15 new international organizations; 104 percent increase in employment of scholarships’ recipients; 20 percent of MSs scholarship recipients (out of 84) considering pursuing a PhD. In 2018, following the Page 17 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) NESP model, MoS launched a new PhD scholarship program and already funded 19 PhD students. The target is to support 50 young researchers over 3 years (about 30 percent of all graduates enrolled in Doctoral programs). There is a good probability that the new MoS scholarship program will help increase the enrollment of (post)doctoral students, while steadily growing, is still low in the country. The newly designed program has also addressed some of the NESP weaknesses. Although greater knowledge and capability to influence the country research agenda was expected from doctoral students, NESP disproportionally supported MSs students, who have many funding opportunities with the ERASMUS + Program. • MoE set up an office for bilateral and multilateral programs to provide more comprehensive system of support for students and faculty, including to support to institutions to actively participate in the EU Erasmus+ Program. HERIC helped the new office build capacity to effectively and efficiently manage mobility proposals across 80 international universities. Among many areas, HERIC, through the Institutional Improvement Grants (IIG) activity, supported a new software purchase for UCG in managing and monitoring mobility applications, which greatly reduced the transaction time: what took half a day to process is now taking a few clicks. Between 2014 and 2018, there were 1,200 mobilities of staff and students from Montenegro and about 450 staff and students coming to Montenegro; both numbers continue to grow. The GoM should now focus on more strategic partnerships and longer-term goals for internationalization beyond general improvement of the quality of studies. • HERIC helped establish a systematic cooperation with the Montenegrin Scientific Diaspora. The project contributed to the development of a Strategy for Collaboration with Diaspora (2015-2018) covering several topics, including education and science, and provided valuable recommendations for better networking and cooperation with the Diaspora. HERIC also helped develop a Diaspora database, as part of the “Science Network” research information system (which the project helped upgrading). The database is being kept up-to-date and actively marketed to attract more scientists, residing abroad. Finally, the project provided financial assistance to the “Montenegro 2017” conference, co-organized by MoS and the Montenegrin Chamber of Economy. The event was focused on advancing networking between researchers and business as well as strengthening linkages with the Scientific Diaspora. As a result of all these efforts, MoS now systematically encourages the Diaspora participation in national and international R&D projects, which should help farther strengthening R&D human capital. • HERIC advanced several incoming mobility initiatives to attract more international students and staff to Montenegrin HEIs as well as improved infrastructure at select UCG facilities. A Feasibility Study on Study Programs in English, funded by the project, identified programs with a strong potential to be offered in English and provided an implementation plan. There are now a few courses at UCG offered in English; and the GoM has allowed for accreditation of English-language study programs. The focus should now be on accrediting at least one full program. HERIC also funded a policy paper with initial recommendations for how to upgrade the existing recognition process of foreign diploma. Based on the paper inputs, MoE established a working group to advance on this matter. Finally, HERIC funded minor refurbishments at 8 UCG forms and helped better equip premises of 4 UCG facilities of individuals with disabilities. Justification of Overall Efficacy Rating 38. The overall efficacy rating is Substantial. All but one PDO indicators have met or exceeded their targets. • The quality and relevance of the higher education sector has been strengthened, accomplished through comprehensive structural reforms of higher education financing and quality assurance systems. This resulted in: (i) the roll-out of the National Quality Assurance Framework, according to which all Montenegrin HEIs started applying new quality assurance and accreditation measures, in line with Bologna-defined EU norms and practices (which embed both quality and relevance characteristics), and made significant advancements towards ESG standards and guidelines, as attested by independent external evaluations; (ii) the implementation of a new HE Page 18 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) funding model in the UCG, promoting more efficient and effective use of public funds as well as research and innovation; and (iii) stronger capacity among MoE and HEIs to monitor and evaluate graduates transition and labor market demand to optimize the offering and quality of HEIs programs. • The quality and relevance of the research sector have strengthened, achieved through the establishment of (i) the first country’s pilot CoE, which provided a framework for future CoEs; (ii) public/ private R&D partnerships with international participants in the areas of national research & development priorities, which produced significant scientific accomplishments; (iii) new funding tools for competitive and innovative research, (iv) upgraded science and technology infrastructure and institutions, and (v) increased collaboration between science and private sectors. • Collaboration between higher education and research sectors have become stronger; internationalization and integration of these sectors within the EU and globally have improved. • The unachieved indicator of the perception of higher education relevance did not impact the accomplishment of the PDO. The lack of achievement is rather a methodological flaw. Many of the implemented changes – such as 25 percent mandatory practical training, new funding model, internationalization and mobility efforts –indicate that even if there is no straightforward way to measure how relevant higher education is, the systematic changes should reduce the mismatch between the skills of higher education graduates and labor demand. C. EFFICIENCY Assessment of Efficiency and Rating 39. Project interventions generated good economic gains and efficiencies. The PAD presented an analysis of the economic benefits of the project for society. The ICR analysis focused on the review of economic gains and efficiencies produced by individual project components. While Annex 4 describes the findings in detail, the most significant ones are the following: • Implementation of a new HE funding model has created efficiencies among public HEIs. The performance contract between the UCG and GoM aims at increasing university accountability, stimulating efficiency and the connectivity between spending and the quality of education. The mandatory 25% of practical education at HEIs has been promoting development of skills that are relevant for the market, therefore, having a direct impact on students’ employability. • The establishment of the NQAF has generated efficiencies in the HE system. The number of HEIs and academic programs were significantly reduced (from 13 to 9 for HEIs; and from 300+ to 208 for the academic programs), while at the same time the public resources were channeled to the most relevant courses from the market perspective. Also, a new study model (composed of 3 years of Bachelor + 2 years of Master + 3 years of PhD studies) was implemented according to the prevailing models in the EHEA and the Bologna process8. This harmonization will likely facilitate international integration, increase participation in the mobility programs as well as cooperation in international research projects. • A new office for bilateral and multilateral mobility programs was set up to facilitate access to international funding for education. The capacities of the new office were built to efficiently and effectively manage mobility proposals across 80 international universities. 8This model also took into account the recommendations made by the European University Association during its review of the entire Montenegrin higher education sector in 2014. Page 19 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) • The Scholarship Program for Master’s, PhD, and Postdoctoral Studies has been already generating economic impact. The program’s evaluation report confirmed the employment increase of MSc and PhD scholarship holders by 104 percent; career progress of all postdoctoral scholarship holders; and a positive impact on participation in international projects. • Establishment of Pilot CoE and funding of eight Collaborative R&D grants enabled implementation of an entirely new approach to R&D funding in Montenegro. The new approach promotes multi-disciplinarity, research quality, market applicability, sustainability, connectivity to international R&D networks and execution of projects in partnerships. The program has created new capacities across Montenegrin HEIs in accessing international funding programs. Both, CoE and Collaborative R&D Subprojects encouraged cooperation between domestic and leading international institutions as well as the Montenegrin business sector. Out of 22 products or services developed via CoE and Collaborative R&D Subprojects grants, 15 were applied in the public or private sector. The program imposed the requirement of contribution (either financial or in kind) by the participating partners. This condition has been transformed into the obligatory financial contribution by the grant holders in all new programs managed by the MoS. Such approach supports economic efficiency and is aligned with international practice. • The project ensured efficient use of the R&D equipment located in the country. Two mapping exercises of the existing research equipment were conducted in order to optimize its use through equipment sharing and avoidance of investment in the same apparatus. 40. The project demonstrated efficiency in design and implementation aspects. The use of international technical experts in areas of financing, quality assurance, and research and innovation helped successfully implement the project’s interventions and build capacity within key stakeholders in the line ministries and HEI. The PMT team, comprised of highly competent staff, established very effective and lean processes for managing grant proposals and engagements with multiple stakeholders. Low staff turn-over on both PMT and Bank teams ensured efficient project management. Savings incurred during the implementation including via VAT exemptions for imported scientific equipment, allowed for the funding of additional tasks relevant to the PDO. Several existing R&D grant recipients received additional funds for their projects, based on the rigorous selection criteria. HERIC also supported the design of the Science Technology Park in Podgorica. 41. The project was extended to accommodate various delays during its implementation. The project closed 27 months beyond the appraisal closing date in order to accommodate the delays during the implementation due to: a) low capacity of HEIs/ RIs to implement CRDS grant proposals, following World Bank procurement procedures; b) the establishment of the CoE as a legal entity; c) the completion of the National Excellence Scholarship Program linked to the two-year academic cycle of the many overseas programs. Most of delays were largely administrative and did not have long-term material impact on the project’s implementation. The decision to extend the project facilitated the government’s efforts to complete all the planned interventions and successfully achieve the PDO. 42. The efficiency is rated as Substantial. The project demonstrated strong efficiency gains across its components as well as efficient design and implementation. The implementation delays were not inefficient but allowed opportunities to complete all the project activities and strengthen the capacity of local actors. Page 20 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) D. JUSTIFICATION OF OVERALL OUTCOME RATING 43. The overall outcome rating is Satisfactory. Relevance of project objectives is High, Efficacy is Substantial, and Efficiency is Substantial. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 44. The project did not specifically seek to address the gender gaps in the education and research sectors. However, the project tracked the participation of women in NESP, CoE, and CRDS programs. In each case, the programs were gender-balanced: out of 132 scholarships, 76 were awarded to women; out of 327 staff involved in research grants, 138 were women. Institutional Strengthening 45. The project had a substantial impact on institutional development. All its components were designed to support the institutional strengthening of MoE and MoS, Quality Assurance Agency, and HEIs to conduct ambitious structural reforms and ensure their long-term sustainability. Key institutional strengthening interventions included: a) external evaluations of HEIs, the tracer study, and institutional improvement grants helped build capacity within MoE and HEIs for systematic self-evaluation, evidence-based decision-making, and longer-term strategic planning and has laid a foundation for continuous improvement of the higher education sector following external independently-defined standards; b) HERIC helped establishing and operationalizing a new national independent Agency for the Control and Quality Assurance of Higher Education; c) CoE and CRDS programs helped elevate the R&D and innovation agenda in the country, resulting in the MoS budget increase by 60 percent and the establishment of new R&D funding instruments and programs; d) HERIC assisted with modernizing scientific and technology infrastructure of Montenegrin RDIs, helping scientists produce higher quality research outputs; d) the project helped built capacity within MoE, MoS, and HEIs for how to administer grant programs and how to apply for competitive external funds. Mobilizing Private Sector Financing 46. To incentivize participation of grant recipients and diversify its sources of funding, the project mobilized financing or in-kind contributions from partners. Institutional Improvement and CRDS competitive grant programs generated additional EU250k, which were mainly used to procure equipment for partners. MoS has continued applying the principle of co-financing in the follow-up initiatives: in the first round of the new annual Innovation Projects Grant Program (2018-2020), 40 percent of funding (EUR440K) was provided by partners. Poverty Reduction and Shared Prosperity 47. The project did not explicitly measure its impact on low-income rural and urban populations and other vulnerable groups or the impacts of poverty reduction. However, improving the quality and relevance of higher education and Page 21 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) research should lead to increased employment of graduates and potentially higher wages, which would contribute toward poverty reduction and have a clearer impact on issues of shared prosperity. The new higher education funding model envisages free studies for undergraduate and graduate students, which should provide equal opportunities for all students of public HEIs, regardless of their financial background, and contribute to shared prosperity. Other Unintended Outcomes and Impacts 48. The project provided small contributions to making access to higher education more inclusive for persons with physical disabilities. HERIC funded infrastructure improvements at four UCG faculties, which has resulted in the refurbishment of eight dorms and better equipment of the premises of four UCG facilities with specific interventions meant to provide greater accessibility for individuals with physical disabilities. The ICR cannot confirm if as a result of this intervention the number of people with physical disabilities accessing higher education at UCG has increased. The project also tackled ‘corruption’ in the higher education sector by investing in software to catch plagiarism. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 49. Project design anchored in the GoM’s reforms program. The project was designed in line with the government’s Higher Education and Research reform priories and was envisioned as an implementation vehicle of the reforms. This ensured a strong government commitment during both preparation and implementation. Both Ministers of Science and of Education were reform champions and provided strong leadership and a clear vision for change, which were detailed in two national policies, adopted before the project start: Strategy for the Development and Financing of Higher Education (2011), Law on Scientific Research Activity (2010). 50. Concise PDO and Results Framework but with a few shortcomings. Although concise and well reflecting the multisector nature of the intervention, the composite and broad nature of the PDO could be interpreted and measured in different ways (a) quality of higher education and research, b) relevance of higher education and research; OR a) quality and relevance of higher education, b) quality and relevance of research, etc.). The ICR interpretation (a) quality and relevance of higher education AND b) quality and relevance of research) was based on the analysis of the design and other evidence. It would have been better to avoid composite PDOs and leave no room for interpretation. The Results Framework is simple and easy to follow; most indicators were well aligned with the PDO. Although a few indicators could have been improved (discussed in more details in the M&E section), the overall results framework provided a clear picture of the project intentions. 51. Simple design and appropriate plan for monitoring. The project design was simple, with clearly structured components, and was based on the country and sectoral context. In addition to well-defined measures to monitor the progress implementation, the design embedded external evaluations by independent international experts to assess the efficacy of individual interventions. These evaluations greatly helped capacity building efforts during the implementation as well as provided inputs into the follow-up programs by the government. This design greatly contributed to the sustainability of the project’s results. Page 22 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 52. Solid analytical underpinning. The design of the project benefited from 1) extensive stakeholder consultations and technical discussions with national, regional, and international experts; 2) global and regional experiences and sound practices, contextualized for Montenegro; 3) lessons learned from the implementation of similar World Bank-financed projects in the world, the Western Balkans region, and in the country, including the following projects: Montenegro Education Reform, Croatia Science Technology, and Chile Millennium Science Initiative. Based on the analytical inputs, HERIC’s design included the following principles: • Prioritizing Quality and Relevance. Successful higher education systems focus on enhancing quality of education and improving links with the labor market. This requires strengthening internal and external quality assurance processes in the higher education system. • Competitive grant funding. Competitive selection process, which is focused on excellence in research and greatest potential to drive quality and innovation of education, we well as international collaboration drive innovation and produce substantial improvements in education and research quality, human capital development, and economic relevance. • Internationalization. A strong international dimension to improve cross-border knowledge and research exchanges is critical in smaller systems. It is achieved through the involvement of international experts in the selection and evaluation of the grants to be supported and integration of recognized international research institutions, as well as international companies, into the R&D projects. • Performance-based contracts. Performance agreements are found to be a highly effective way to promote innovative approaches to research and change norms and drive efficiencies in HEIs. 53. Insufficient donor coordination. To avoid duplicative activities in the design of the project, the Bank team closely coordinated with donor representatives in the country. However, the level of coordination between local representatives and technical counterparts in their respective headquarters was over-estimated. Given the prominence of the EU accession discussion in the country and in Europe, the team could have sought closer coordination directly with relevant technical agencies in Brussels. The EU proposal to fund the development of the National Qualifications Framework, which was envisioned as one of the project’s deliverables, came as a surprise to the Bank team early into implementation when the alternate funding was accepted by the GoM. 54. Adequate risk and mitigation measures, but some risks underestimated. At the time of appraisal, the overall risk rating was Moderate. Although most critical risks and related mitigation measures were clearly assessed during the design phase, a few were underestimated. The risks of absorption and implementation capacity of HEIs and RIs, including the ability to comply with World Bank procurement procedures, materialized during the project implementation and led to delays. The complexity of an institutional set up of an CoE, including its legal status, emerged as risk, which was not envisioned during the design and led to delays. 55. High readiness for implementation. Strong government commitment, capacity building exercises during the project preparation, detailed proposals and instructions for grant programs, and a competent PMT team assembled prior to the project implementation ensured high readiness and a quick start after the effectiveness. B. KEY FACTORS DURING IMPLEMENTATION Page 23 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 56. Highly-impactful coordinating and engagement function by the Project Management Team (PMT). The role of PMT proved particularly important given that the project included two implementing ministries, MoE and MoS, and multiple stakeholders (all HEIs, many grant recipients for subprojects, etc.). PMT provided very effective and efficient oversight and guidance across multiple interventions. The team also played a critical role in building capacity, facilitating exchange of experiences and information, creating high standards for transparency and accountability, and building awareness of the structural reforms supported by the project among key stakeholders and the society overall. 57. Strong Government commitment reinforced by legislation. Both MoE and MoS were highly committed to the project and strategically drove the implementation of reforms in quality and relevance of higher education and research and innovation through HERIC. With the appointment of Sanja Damjanović as Minister of Science in 2016, the GoM showed its deepening commitment to scientific research, since Minister Damjanović is a world-renowned physicist affiliated with the European Organization for Nuclear Research (CERN) in Switzerland. Minister Damjanović has promoted conferences, partnerships, and high-impact regional initiatives such as the South East European International Institute for Sustainable Technologies, none of which would likely have been possible without the foundations set by HERIC. Many of the recommendations from external evaluations funded by the project were reinforced by subsequent legislation and formed the basis of national strategies and policies developed during HERIC implementation (see the list in the efficacy section). The strong relationship between the World Bank and the ministries helped address any outstanding issues and follow up in a short time. The recommendations highlighted in the Aide Memoires would be promptly put into action. 58. Minimal staff turn-over rate. There were very few staff changes among key government counterparts and PMT; while the Bank team composition remained the same over the project duration. This ensured smooth implementation and established a strong trusted partnership between all involved actors. 59. Effective use of the M&E system. Adherence to the M&E plan with proper baseline and follow-up implementation ensured availability of reliable data. The PMT provided very granular systematic reporting and evidence collection on achieved milestones and results during the implementation. The results of impact evaluations on various project’s interventions as well as real-time international expertise were used during the implementation to further enhance and improve the project’s outputs as well as to design follow -up government programs. 60. Weak capacity of key actors. • All but one grant project in the CRDS program experienced delays in implementation, requiring additional time (between 4 to 17 months) to successfully accomplish proposed experiments. The delays were primarily caused by: a) overly ambitious timeline; b) lack of experience of project teams in complex administrative procedures, mostly due to the lack of experience with World Bank procurement. Special training in procurement and related procedures before the roll-out of grant projects, which all grant recipients received, did not fully address all the capacity gaps. The Bank hired an external procurement specialist to boost the implementation capacity; this helped addressing many of the implementation bottlenecks. While the HERIC PMT was assessed as being very effective and helpful, the CRDS impact evaluation report notes lack or very poor institutional support of grant recipients’ academic institutions, in particular UCG (which was going through many organizational changes and new reforms at the same time). This also caused delays. Page 24 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) • Like the CRDS program, the BIO-ICT program, implementing the CoE grant, struggled to fully absorb all the funding. This not only related to the large grant size but also to a complex multi-disciplinary engagement which needed to be better defined and many nascent innovation concepts being piloted through HERIC for the first time in the country. Finally, it took longer than expected to define the legal status of the CoE to ensure it is converted from a program to an institution. • It also took longer than expected to implement the National Excellence Scholarship Program due to: 1) insufficient experience in developing such programs in the past, including the proper design and communications campaign; and 2) the complexity of the program as being joint between MoE and MoS. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 61. The project’s M&E was relatively well designed and aligned to the development objectives. The M&E of the project involved: a) performance indicators; b) quarterly and at times of high complexity and intensity of the implementation monthly progress reports; c) a midterm review of implementation and outcome progress; d) impact evaluations and external expertise. The results framework was simple. It identified three areas, corresponding to project components, that were related to the PDO, with a total of four indicators at the PDO level and eight at the intermediate level. The results framework included baselines and targets for all indicators (except for the perception of relevance indicator, see below) and overall provided a useful summary of the project. Many of the supplemental indicators were included in the administrative M&E system, managed by the PMT (e.g., training, grant project outputs, type of research infrastructure upgrades, academic outputs, etc.). The use of impact evaluation studies and external expertise during the project implementation was a powerful feature of the M&E design, which ensured continuous feedback loop on projects interventions, stronger buy-in by stakeholders, opportunity for course-corrections, and a very early adoption of project results in the follow-up programs by the Government. 62. Some results indicators could have been better defined and updated. • The “Complete external evaluation of three main universities, utilizing new quality assurance and accreditation measures developed in accordance with Bologna-defined EU norms and practices” PDO-level indicator was composite. It could have been broken-down into separate indicators (the latter was reflected in the in the theory of change). • It was justified to include the perception of relevance PDO-level indicator as an attempt to better define relevance in higher education. In the absence of well-defined indicators in this area in the sector, the team clearly indicated an ambitious attempt during the project to develop a methodology for how to measure relevance. However, it would have been more prudent to define the target once the baseline was determined. And once the relevance index was measured and found not to be useful, the team should have put more efforts in defining an alternative indicator. The choice of the indicator remained suboptimal until the project closure and required a supplemental evidence from the ICR to confirm the PDO advancement. • Given the prominence and impact of the intervention, the funding model intermediate indicator would have been better placed at the PDO-level (reflected as such by the ICR in the Theory of Change). • The implementation of the National Qualifications Framework was done by another agency. The intermediate indicator to measure it should have been dropped. Page 25 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) M&E Implementation 63. The PMT demonstrated great competency in regularly collecting and evaluating the progress of project activities. The PMT was responsible for day-today M&E activities, including collecting information from line ministries, tracking project implementation progress, and monitoring the results indicators. In addition to the project Results Framework agreed to with the Bank, the PMT used a more detailed M&E system with supplemental indicators, templates and well-established frequency of reporting for grant recipients and other stakeholders. 64. The Bank team missed the opportunity during the restructuring to adjust the results framework. A few indicators could have been improved, per earlier discussion on the M&E design to strengthen the results framework. The target for the number of research and development subprojects intermediate indicator was lowered from eleven to eight to reflect the fewer than anticipated but bigger in size grants awarded. The target change did not impact project account; however, it should have been updated in the results framework. The Bank team could have kept more up- to-date the actuals of the R&D grants indicator in the ISRs; this was fixed in the final ISR. M&E Utilization 65. The project did a very strong job at utilizing the data and knowledge produced through the M&E system and played an important role in establishing the country’s M&E systems. M&E was utilized to: a) monitor and manage project progress, particularly through intermediate indicators and supplemental indicators from the PMT; b) identify issues and course correct (e.g., bringing an external procurement expert to speed up the implementation of R&D grants); d) provide a basis for such important decisions as project extensions. The project was very effective at utilizing external expertise and impact evaluations to build capacity among key stakeholders to self-evaluate and identify and implement evidence-based strategic action plans and policies. This approach helped solidify stakeholders’ buy-in for ambitious reforms and ensured their strong participation in the implementation. The GoM also designed follow-up programs based on the results of impact evaluations of HERIC’s interventions (e.g., new PhD scholarship progr am, Innovation Projects Grant Program) and applied stringent M&E procedures, developed by the PMT (e.g., use of external experts, detailed assessment criteria, etc.). Justification of Overall Rating of Quality of M&E 66. The quality of M&E was Substantial. The overall design and implementation of the M&E system was suitable, the project’s impact evaluations were of great value; and the implementers did proper collection and reporting of data and a meaningful use of the system to create tangible improvements to the project. The shortcomings included a few deficiencies in the results framework which could have been but were not addressed. However, supplemental indicators, managed by the PMT, offset the deficiency. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 67. Environment. The project was categorized as environmental safeguards category B. It triggered Operational Policy OP 4.01 on Environmental Assessment due to financing of: a) minor construction works related to renovation of student dormitories; and b) R&D grants. An Environmental Management Framework (EMF) was prepared in November 2011 and completed with disclosure and public consultations, held in Podgorica on Page 26 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) October 21, 2011. All project activities were subject to environmental screening and followed Montenegrin law, World Bank safeguards, and the project EMF. Environmental Safeguards Semi-Annual Progress Reports were prepared regularly and reviewed by the Bank. The PMT was staffed with a part-time environmental specialist, who provided exceptional guidance to teams and ensured adequate implementation of the due diligence. All environmentally related activities under the project were successfully implemented. The Final Environmental Management Plan Compliance Report confirmed that there were no complaints received by the PMT from the general public. 68. Procurement. The fiduciary responsibilities of the project were carried out by the Technical Service Unit (TSU) established within the Ministry of Finance and supported by the PMT. Procurement packages were relatively small and subject to post review; Bank’s recommendations were diligently addressed. The TSU staff followed Bank Policy and Guidelines, with an overall satisfactory performance. Although the project experienced delays, among other reasons, due to weak capacity of grant recipients to follow World Bank procurement guidelines, no critical issues from procurement prospective were noted. 69. Financial management (FM). FM performance was assessed as satisfactory throughout the implementation period, from the aspects of planning and budgeting, accounting, reporting, internal controls and flow of funds. Adequate implementation capacity has been maintained during the implementation by the TSU. Legal covenants related to quarterly reporting and audit were complied with, however the audits were delivered typically with two to three months delay after the due date. Given that audit opinions were clean (unmodified), i.e.: the audits did not identify any issues, it was assessed that lack of timeliness did not pose such a significant threat to downgrade the FM performance rating. C. BANK PERFORMANCE Quality at Entry 70. The quality at entry is rated as Satisfactory. The project design was highly relevant to national priorities and had strong buy-in and champions from both Ministries of Science and Education; both institutions planned to use the project to implement ambitious reforms in higher education and research sectors. The preparation team had an adequate mix of skills to design a multi-sectoral intervention. The PDO reflected the country’s development goals; however, its composite nature created ambiguity in the interpretation – which was clarified in the PAD narrative and follow-up ICR interviews. The scope of the project was clearly defined. The design was straightforward and informed by thorough consultations with key stakeholders and experts, global and local best practices, and lessons learned from similar interventions. It was also well adjusted to the country context. The donor coordination with EU agencies could have been done beyond local representatives and could have included Brussels-based agencies to avoid duplicative efforts. Nevertheless, this did not affect the PDO achievement. The procurement, financial, and environmental arrangements were adequate. The M&E system was suitable. Although some indicators could have been improved, the Results Framework was concise, complete (baseline, targets, data sources), and captured well the nature of the project’s interventions. Exception for perceived relevance all indicators were measurable. The M&E approach offered a few best practices which could be helpful in similar interventions in other Bank projects: a) the use of external evaluations early in the project to create a common understanding of the issues by HEIs, REIs, and the ministries and constructively engage them in implementing the necessary changes; b) impact evaluations to provide early feedback to key stakeholders on if and how to design of follow-up programs. Although the risk assessment was candid and proposed adequate mitigation measures, it also contained a few gaps. The risk of absorptive capacity was Page 27 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) underestimated. Hence, the proposed implementation timeline was overly ambitious. The legalistic challenge of establishing an CoE as an institution is not uncommon and could have been provisioned based on a more thorough analysis of similar interventions in other projects. A strong relationship with the government and capacity building exercise during the preparation ensured a high implementation readiness, including a highly competent client implementation team (PMT), and no delays at the project start. Quality of Supervision 71. The quality of supervision is rated as Satisfactory . The project’s successful implementation - timely effectiveness, early disbursement, course correction when needed, and accomplishment of all intended results - was anchored in strong Bank supervision. The Bank team worked closely with the PMT, the ministries, and HEIs and conducted regular supervision missions, followed up by clear progress reports, valuable timely feedback to counterparts, and technical assistance to ensure successful implementation of the project. HERIC benefited from a good multi-sectoral mix of skills on the team and no staff turn-over during preparation and implementation, which greatly contributed to the strong partnership with counterparts and stakeholders and quality of supervision. The use of external evaluations early in the project as well as external expertise and impact evaluations over the duration of the project ensured continued stakeholder commitment, the execution of complex reforms, and a good indication of sustainability of results through follow-up programs by the government. The Bank team missed the opportunity during restructuring to adjust a few problematic results indicators and could have kept more up-to-date the actuals of the R&D grants indictor; the latter was fixed in the final ISR. The team could have also put more effort in better defining how to measure relevance of higher education. Most risks were well anticipated and prepared for, except for the low absorptive capacity risk which was underestimated, materialized during the implementation, and led to delays and extensions. However, through effective technical assistance, the Bank team helped contain and mitigate the issue. The risk of delays in the institutional CoE establishment due to legalistic matters was not anticipated, emerged during implementation, but was eventually solved with the support of the Bank team. HERIC followed fiduciary and safeguard policies. Justification of Overall Rating of Bank Performance 72. The Bank’s performance was Satisfactory. The project has a few minor short-comings in design – in the formulation of results indicators and an overly-ambitious timeline. Nevertheless, this did not prevent the achievement of development outcomes. D. RISK TO DEVELOPMENT OUTCOME 73. The risk to development outcomes is modest. The risk varies by outcome. The risk to sustainability of higher education outcomes is modest. The government has continuously supported the ambitious reforms and, together with HEIs, showed commitment throughout project lifetime at achieving and maintaining main outcomes. The project helped significantly strengthen the country’s systems. New legislations and a new independent QA agency have enforced improvements in the quality and relevance of higher education. The systematic (re)accreditation process, based on external independent evaluations, following stringent ESG standards, is mandatory to all HEIs. A non-compliance results in a loss of license and a public communication about it. It is a strong incentive for HEIs not to fall back on new standards. The new funding model and the Page 28 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 25% practical training requirement will likely increase the relevance of higher education to potential employers. The student outgoing mobility has been increasing, according to the European Commission. MoE is now actively improving the inward mobility, by preparing academic programs in English. One area of concern is how UCG will be coping with the loss of revenue. The new funding model and legislation instituted free admission for all students while limiting tuition to cases of poor performance. Despite of state budget increases, UCG has lots a substantial funding source. The government strategy is to incentivize faculties to apply for competitive grants for additional revenues. HERIC significantly invested into building capacity of HEIs to do that as well as institutional support (e.g., an office to coordinate mobility program in MoE). However, the strategy is risky and highly depends on the capacity of staff to compete for external funds and ability to prioritize between core job functions and search of funds. At the same, the GoM’s EU accession strategy will continue prompting MoE not only to sustain but to deepen the reforms. 74. The risk to sustainability of research outcomes is closer to substantial. The GoM has championed sweeping reforms in the sector, which was not as mature compared to higher education at the project start and hence required (and continue requiring) more efforts to change. With project support and strong push by MoS, the country’s competitive research and innovation environment was established, where many new concepts were successfully piloted. The sustainability of many project achievements is supported by new legislations, strategies, and streamlined into new programs. A strong government dedication to reforms is also demonstrated in the 60 percent increase of the MoS budget. However, despite the budget increase, state research funding is still low compared to other countries in the region and peers in the EU9. To address the challenge while also continue boosting quality and relevance of research, the government strategy is to rely on external funding. It is risky and may result in more stress among HEIs/REIs. The government has also predominantly focused on strengthening upstream R&D systems as perceived to be the largest drivers of innovations. It is equally important to invest into the competitiveness and innovation of the private sector. The GoM is aware of it and has already requested a new loan the Bank to support jobs and growth in the country with the focus on strengthening competitiveness and innovation of the private sector. The government may not have the adequate capacity to maintain and depend sector reforms while embarking on the new ambitious agenda, the success of which is critical to the research sector. It will be very important to stay focused on strengthening the human capital, keep realistic expectations, and follow the principles and framework, created by HERIC. The government EU accession strategy will be a strong anchor to support and sustain the existing and new reforms. V. LESSONS AND RECOMMENDATIONS 75. External impetus for reforms is a powerful driver of changes, especially for small states. In countries like Montenegro, which is heavily depended on regional integration for its social, political, and economic welfare, EU standards and guidelines, including for areas like accreditation, research, and mobility of staff and students, provide both the frameworks for internal reforms and the credibility behind those reform initiatives. This was especially true for HERIC, which had the Bologna standards and requirements for EU accession as background rationales for the design of the components of the project. By utilizing such a globally and regionally respected framework in the design of the project, the ministries and the Bank teams were able to 9 OECD average in 2016 was 2.34% GDP, while Montenegro was at 0.32% of GDP. Page 29 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) develop a sound project with strong stakeholder ownership and impact on national policy goals outside the sector. • Recommendations: a) Assess and understand the local policy context to include factors outside the immediate sector. 76. Use of external evaluations throughout the project as part of the M&E system is an effective approach to facilitate reforms. External evaluations by an independent reputable international agency early in the project were a powerful way to build a common vision of issues and solutions among stakeholder and keep them in the driving seat at the implementation of the necessary changes. The project found an exemplary approach to capacity building among multiples stakeholders. Many of them attested during the ICR interviews that external expertise and learning by doing were the biggest accomplishments of the project. • Recommendations: a) Include external evaluations by international experts in project design to set baseline, periodically assess progress and adjust plans based on information compiled; b) Plan on using it early in the project. 77. The time and effort required for deep structural reforms and development of the country’s human capital should not be underestimated. The implementation of complex structural reforms, including efforts to develop a knowledge-based society, takes significantly more time that the five years of implementation of a typical IBRD-funded project. Carefully managing expectations and the scope of the reforms is paramount. Planning for hands-on support to deal with burdensome administrative requirements and ensuring the availability of additional expertise during the implementation of grant proposals may help lowering the risk of delays and reduce the stress of recipients over their inability to follow complex and stringent funding requirements. • Recommendations: a) Invest in substantial capacity development for establishing national systems in quality assurance and quality culture and research and innovation; b) Make provisions at project design for extensive direct, one-on-one capacity-building activities and resources during the implementation. 78. Labor Market Relevance is a challenging concept to measure in a project’s timeframe. The use of a labor market relevance index indicator to establish a baseline and measure impact of the project was conceptually sound but extremely difficult to execute with confidence. Revising the results framework at MTR to preempt potential challenges to measuring relevance when the baseline came in quite high was considered, but no better indicator was found at that time. • Recommendations: a) Consider indicators that are more immediately quantifiable, such as programs for internship training and engagement of employers in the curriculum development; b) utilize the MTR opportunity for restructuring to pursue purposeful changes where ambiguity exists in the RF. 79. To ensure sustainability of the results a special focus should be placed on strengthening inter-ministerial collaboration and development of a sustainable plan for future funding of all components of the National Innovation System (NIS). The Ministry of Science, Ministry of Economy, and Ministry of Education all play a role in designing and implementing policies for innovation and human capital development. There is a space to improve coordination between these efforts, to avoid inefficiencies and to ensure innovation development. Creating a dedicated Innovation Agency, as well as an Innovation Coordination Council, could improve program implementation and coordination and could serve as an implementation unit for the Smart Specialization Strategy in Montenegro. Page 30 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) • Recommendation: a) Consider creating a specialized Innovation Agency/Fund which allow for more professional and sustainable implementation of innovation support mechanisms, including serving as a platform for dialogue between the innovation system stakeholders and public-private consultations on innovation programs, joint projects and business sector needs. 80. Competitive grant funding is a powerful driver of innovation, but it requires strong supporting M&E systems to be effective. The competitive process encourages potential grantees to develop a project with carefully thought-out goals and objectives, methodology, personnel, and a budget. This often leads to substantial results - be it in the field of science, education, or an institutional strengthening effort. A strong M&E system needs to be designed to support competitive programs, including a transparent and rigorous methodology (including external independent peer reviews), procedures, and principles. • Recommendations: a) Broaden the use of competitive grant funding in strengthening institutions; b) Strengthen the central capacity to monitor the implementation progress of grant support activities and grant recipients hosting institutions. 81. Lessons from CoE and other R&D grants programs in weak-capacity environments: • More narrowly focused grant proposals and letting the “more flowers bloom” approach could be a more effective way to increase the ability of local actors to absorb funds. The size of grants in the CRDS and CoE programs potentially should have been smaller. Two smaller-scale CoEs could have been piloted instead of one. MoS has considered this lesson by designing the follow up programs with smaller grant sizes and envisioning a few more CoEs in the next round of proposals. • Phased funding approach to scale-up or strengthen the achievements of more successful projects may be more effective. Each phase should follow stringent criteria and competitive principles. • Better support to the grant recipients by the management of their host institutions is not guaranteed and may lead to delays in implementation. Calls for competitive funding could incorporate control mechanisms for institutional support which can be part of the evaluation process. Line ministries should also systematically and frequently monitor the institutional support for funded grant projects. • When establishing an CoE which is expected to be sustained beyond the grant funding period, it is critical to clearly define an institutional framework (including the legal status, operating model funding, and ownership) before then grant project starts. Institutional framework should be specified either in grant proposals and be part of the evaluation process or be a condition for funding allocation. • To promote R&D commercialization and private sector investment in university R&D, it is essential that HEIs develop its own “commercialization agenda.” Such an agenda shall clarify the process of creation (and IP ownership) of technology-based start-ups spinning-out from research departments. • The sustainability of even most successful grant projects is not guaranteed. Involving local organizations, including municipalities, may streamline the efforts beyond grant funding and benefit the local society. • Recommendations: a) Keep grant design simple and start small; b) Design grants for scale with an ability to add more funds later based on results; c) Strengthen the enabling environment of grant recipients host institutions; d) Focus on the institutional framework before establishing an CoE; e) Careful select partners to ensure sustainable results; engage local municipalities and organizations; f) Ensure that the development of the university commercialization agenda is a priority for the HEIs management . Page 31 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Strengthen the quality and relevance of higher education and research in Montenegro Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Complete external evaluation Text 0 3 4 of 3 main universities, utilizing new quality 20-Dec-2011 20-Dec-2011 10-Jun-2019 assurance and accreditation measures developed in accordance with Bologna- defined EU norms and practices. Comments (achievements against targets): As a result of the project, a National Quality Assurance Framework (NQAF), according to Bologna-defined EU norms, was developed and operationalized: - 2 cycles of external evaluations of 3 universities conducted (baseline, follow-up) - Quality assur ance and accreditation measures applied nationally to all higher education institution - 9 higher education institutions (4 un iversities and 5 independent faculties) (re)accredited. Page 32 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) To clarify further in terms of the number of HEIs – in Montenegro there used to be 3 universities (1 public – University of Montenegro/ UCG and 2 private universities – University Donja Gorica and Mediterranean University) and 10 independent faculty units (1 public and 9 private). In 2017, 5 of the independent faculty units merged into another university (Adriatic University), so now there are 9 HEIs in total – 4 universities and 5 independent faculty units. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increase in the perception of Percentage 0.00 86.00 84.00 relevance of higher education programs and degrees 31-Jan-2013 20-Dec-2011 10-Jun-2019 Comments (achievements against targets): The baseline was to be determined during the project implementation. The 2013 baseline and 2018 follow-up surveys on the perception of relevance (or utility) of higher education by employers in Montenegro did not produce statistically different results. With a highly favorable baseline (the relevance index of 86 out of 100), both surveys concluded that there was not a big gap between the needs of employers and educational outcomes and most employers found university education very relevant. The final index figure came in at 84, which is slightly below the benchmark result from the first round survey. The firm conducting the survey and indexing deemed the final result statistically insignificant, however, particularly in light of the very favorable baseline. The firm also found that the increased visibility and prominence of higher education in Montenegro as a result of the HERIC project likely influenced responses to the questions, resulting in higher expectations for the sector in this round. All these finding are clearly presented in the final report. Nonetheless, there is clear anecdotal indication that the overall context for assessing higher education in Montenegro has been improving, with the sector seen in high Page 33 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) regard. The challenge of the statistical assessment of relevance has contributed to the difficulties in measuring this with accuracy. Although the index of relevance of higher education among employers , introduced in perception surveys, was not particularly useful, the exercise yielded helpful recommendations, including the need for: systematic engagement of employers in curricula development; in cooperation between HEIs and employers, provision of opportunities for practical training; and plans for university enrollment based on the labor market demand. Many of these recommendation to strengthen the relevance of higher education were implemented by the government. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Establish one pilot Center of Number 0.00 1.00 1.00 Excellence 20-Dec-2011 20-Dec-2011 10-Jun-2019 Comments (achievements against targets): As per the project design, the establishment of the pilot Center of Excellence was achieved through funding provided via a national grant, also captured in the intermediate indicator "number of research and development subprojects implemented." Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Establish 10 public/private or Number 0.00 10.00 20.00 international R&D partnerships and/or business 20-Dec-2011 20-Dec-2011 10-Jun-2019 start-ups with research institutes or university Page 34 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) faculties. Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Higher Education Finance Reforms and Implementation of Quality Assurance Norms Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Implementation of new Text Funding model not New funding model New funding model higher education funding developed implemented implemented model. 20-Dec-2011 20-Dec-2011 10-Jun-2019 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of grants awarded Number 0.00 20.00 16.00 to faculties, programs, and institutions. 20-Dec-2011 10-Jun-2019 10-Jun-2019 Comments (achievements against targets): Page 35 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) The project originally envisioned to support 20 grants. However, the final decision was to support slightly fewer, but more substantial interventions, based on the applications received. The impact of the grants, even with the smaller total number, has not been effected negatively by this. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion National Qualifications Text Baseline competencies NQF is being NQF is being Framework developed and not developed for any implemented implemented via adopted for 180 programs program support outside the HERIC project. 20-Dec-2011 10-Jun-2019 10-Jun-2019 Comments (achievements against targets): Numerous other Quality Assurance Initiatives and Activities were pursued to complement the NQF efforts, including - QA stan dards and procedures developed - Tracer study system piloted - Plagiarism prevention framework established and operation alized Component: Human Capital Development through Internationalization Initiatives Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Establishment of an Office Number 0.00 1.00 1.00 Page 36 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) within the MoE to coordinate 20-Dec-2011 20-Dec-2011 10-Jun-2019 bilateral and multilateral agreement programs available to Montenegrin students and academic staff Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of students Number 0.00 60.00 132.00 receiving scholarships for master’s, doctoral and 20-Dec-2011 20-Dec-2011 10-Jun-2019 postdoctoral studies abroad. Comments (achievements against targets): 132 scholarship contracts signed-first and second call. Completed 126 scholarships (82 Masters,31 PHD,13 PostDoc.). Out of 86, 82 Masters scholarships were fully completed by the end of the project; 2 were cancelled by the beneficiaries; and 2 were discontinued as the completion of their studies was delayed due to force majeure. Component: Establishing a Competitive Research Environment Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 37 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Number of research and Number 0.00 12.00 9.00 development subprojects implemented. 20-Dec-2011 10-Jun-2019 10-Jun-2019 Comments (achievements against targets): During implementation, larger projects, with more diverse partnership potential, were awarded the majority of the research funding, resulting in fewer, but larger subprojects: 8 R&D grants under the Program for Collaborative R&D Grants (CRDS) and 1 R&D grant to establish one Center of Excellence. The impact in terms of multi-institutional, multi-disciplinary engagement was as great, if not greater, than anticipated during the project design stage. This calculation is somewhat misleading, as the number of institutions implementing projects is, in fact, 24 (20 from the national grants and 4 via the Center of Excellence funding), but the number of specific projects is 9, including the Center of Excellence. In the end, the decision to award jointly lead or implemented projects between numerous centers under a single grant resulted in fewer but larger projects being funded. As the grant scheme was designed for proposals between US$150,000-US$400,000, the projects selected were all close to the maximum of US$400,000, allowing for larger, more impactful projects/subprojects. So, in counting transfers only, 9 designated accounts were created. But, 24 institutions were involved in these 9 projects, and their work could reasonably be considered to be "subprojects." Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increase in the number of Number 200.00 235.00 585.00 Montenegrin researchers who participate in 20-Dec-2011 20-Dec-2011 10-Jun-2019 international R&D projects. Page 38 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Development of framework Text Framework for future For future CoE For future CoE for future CoE. CoE not developed developed developed 20-Dec-2011 20-Dec-2011 10-Jun-2019 Comments (achievements against targets): Page 39 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Higher education has better quality and is more relevant 1. National Quality Assurance Framework (NQAF), according to Bologna-defined EU norms, is operational: 1.1. 2 cycles of external evaluations of 3 universities conducted (baseline, follow-up) 1.2 Quality assurance and accreditation measures applied nationally to all HEIs Outcome Indicators 1.3. 9 HEIs (4 universities and 5 independent faculties) (re)accredited 2. New HE funding model implemented 3. 20% increase in positive perception of HE programs/ degrees among employers 1. NQAF established, including QA and accreditation measures and institutional capacity Intermediate 2. New HE funding model developed Results Indicators 3. 16 of grants for institutional improvement of HEIs awarded 1. List of accredited and licensed HEIs in Montenegro published on the website of the Ministry of Education and QA Agency 2. The accreditation/reaccreditation documents and procedures listed on the QA Agency’s website 3. Strategy for Development of Higher Education, 2016-2020 Key Outputs by 4. Law on Higher Education, adopted in June 2017 Component 5. Law on Academic Integrity, adopted in March 2019 (linked to the 6. Tracer Study Report achievement of the 7. Plagiarism Prevention Framework Feasibility Study Objective/Outcome 8. 1 workshop on the use of iThenticate software conducted (plagiarism prevention) 1) 9. Report on Funding Mechanisms for the Development of Higher Education in Montenegro 10. Draft Performance Contract 11. 50 events organized 12. visited events 13.Obtained certification of ISO 9001 Quality Management System at the Maritime Faculty, UCG Page 40 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 14. International certification for harmonized curricula in undergraduate study programs at the Maritime faculty 15. The Communication Strategy and Communication Action plan for the UCG prepared and adopted 16. "Virtual enterprise" syllabus developed, courses held for students of the Faculty of Economics, UCG 17. Strategy for QA at UDG with Roadmap for its implementation; established QA Unit at UDG 18. Strategy for reforming and improving Doctoral Studies at the University of Montenegro 19. Revision of the existing curriculum at the Faculty for food technology, food safety and ecology (UDG) 20. PhD research hub established at UDG. The rulebook for using the hub and for protection of IPRs at UDG and technology transfer is created and approved by UDG Senate. Objective/Outcome 2 Research has better quality and is more relevant 1. One pilot CoE established Outcome Indicators 2. 20 public/ private or international R&D partnerships or start-ups with RIs established 1. 9 R&D subprojects implemented, including the CoE Intermediate 2. Increased # of Montenegrin researchers participating in R&D projects Results Indicators 3. Framework for future CoEs developed 1. 11 new laboratories created 2. 14 existing laboratories improved 3. 177 training events organized Key Outputs by 4. 238 training visited events Component 5. 327 Staff involved in CRDS and CoE projects (189 male and 138 female); out of which 236 researchers (linked to the 6. 11 PhD theses completed during the project lifetime achievement of the 7. 16 PhD theses in progress Objective/Outcome 8. 17 MSc theses completed during the project lifetime 2) 9. 7 MSc theses in progress 10. 10 patents obtained/applied for 11. 368 published scientific papers; out of which 112 on SCI list Page 41 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 12. 22 products/services developed’ out of which 15 applied in public/private sector 13. Evaluation of the CoE 14. Strategy for Development of Higher Education, 2016-2020 15. Strategy for Innovation Activity 2016-2020 with the Action plan 16. Strategy for Scientific Research Activity 2017-2021 17. Smart Specialization Strategy 2019-2024 18. Law on Innovation Activity adopted in 2016 19. Impact Assessment of implemented collaborative R&D grant projects. 20. Decision by the Government of Montenegro on establishing UCG CoE as a legal entity "Institute - Centre of Excellence for Research and Innovation". 21. Report on advisory services on intellectual property and academic IP commercialization. HERIC project. 22. The Revised Roadmap for research infrastructure in Montenegro for 2019-2020, Ministry of Science Report on scientific equipment in Montenegro Crosscutting interventions, contributing to both outcomes: Improved mobility and connectivity for students, faculty and researchers, including scientific diaspora Intermediate 1. Office in MoE for bilateral and multilateral programs established Results Indicators 2. 132 of students received scholarships for master’s, doctoral & postdoctoral studies abroad 1. National Excellence Scholarship Program established 2. Office in MoE for bilateral and multilateral programs established Key Outputs by 3. 8 dorms refurbished, premises of 4 UCG facilities better equipped for individuals with disabilities Component 4. Scholarship Management Handbooks for 2015 and 2016 (linked to the 5. Information about mobility opportunities published on the MoE’s website and www.europasscrnagora.me achievement of the 6. Analytical study on the Montenegrin Scientific Diaspora Objective/Outcome 7. www.naucnamreza.me - a platform facilitating networking and cooperation between diaspora and 2) scientists 8. Evaluation of the National Excellence Scholarship Program Page 42 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Roberta Malee Bassett Task Team Leader(s) Nikola Kerleta Procurement Specialist(s) Aleksandar Crnomarkovic Financial Management Specialist Andrea C. Guedes Team Member Amy Evans Social Specialist Kateryna Timina Team Member Sreypov Tep Team Member John Gabriel Goddard Team Member Bojana Naceva Team Member Sanela Ljuca Team Member Julie Rieger Counsel Danijela Vukajlovic-Grba Team Member Supervision/ICR Roberta Malee Bassett Task Team Leader(s) Manjola Malo Procurement Specialist(s) Aleksandar Crnomarkovic Financial Management Specialist Boba Vukoslavovic Team Member Esma Kreso Beslagic Environmental Specialist Dragana Varezic Team Member Sanela Ljuca Team Member Bojana Naceva Team Member Page 43 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Iwona Maria Borowik Team Member Patrick A. Biribonwa Team Member Denis Mesihovic Team Member Daria Lavrentieva Team Member Nikola Kerleta Team Member Andrea C. Guedes Team Member Agnes I. Kiss Social Specialist B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY11 32.355 132,002.15 FY12 29.866 183,785.81 FY13 .800 954.56 Total 63.02 316,742.52 Supervision/ICR FY13 15.038 63,239.88 FY14 11.866 84,453.82 FY15 14.127 83,037.05 FY16 11.154 49,318.24 FY17 21.617 105,876.79 FY18 10.192 80,865.37 FY19 11.102 97,081.48 Total 95.10 563,872.63 Page 44 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Higher Education Finance Reforms and 3.73 1.73 46% Implementation of Quality Assurance Norms Human Capital Development through 2.81 3.30 117% Internationalization Initiatives Establishing a Competitive 7.94 6.47 81% Research Environment Project Management and Monitoring and 1.26 1.19 94% •Evaluation Total 15.70 12.69 81% Comments: The loan was fully disbursed. The difference in approved and actual amounts is due to the different conversion rates applied. The World Bank Loan Department applies different rate each time a withdrawal application is submitted. Given the volatility of the USD, the difference can be substantial. Page 45 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) ANNEX 4. EFFICIENCY ANALYSIS An economic and financial analysis at the appraisal was justified based on international practice concentrated on increasing the quality and relevance of Higher Education. The project has generated economic value in each project component, namely the HE finance model; HE quality reforms; HE internationalization; and the establishment of a competitive research environment. A. Efficiencies promoted by the implementation of a new HEIs funding model As highlighted in the PAD, one of the more significant trends among OECD countries was the shift to HE allocation mechanisms that are performance-based, thus providing opportunities for both efficiencies and modernization of the higher education institutions in Montenegro. In result of the project, a new performance-based financial contracting model was launched from the academic year 2017/18 to improve the quality of public funding. The model was prescribed by the 2017 Law on Higher Education and was adopted by the UCG in December 2018. The goal of the new funding model was to ensure efficient public allocations in line with national priorities, respectively: (1) A performance contract between the UCG and GoM is focused on monitoring the university's efficiency and effectiveness with detailed criteria, standards, and performance indicators in funding and obligations of both GoM and UCG. The model stimulates connectivity between spending and the quality of education. The budget allocations with detailed breakdown-cost are assigned towards well-defined strategic plans than can be traced during the budget implementation (this compares to the lumpsum funding based on staff salaries in the past). (2) A cap on student enrolment was prescribed to improve the student-teacher ratio. (3) Free studies were instituted to promote the quality of learning outcomes10. The new model is aligned with the prevailing models in the OECD economies supporting the provision of equal opportunities for all students of public HEIs, regardless of their financial background. (4) In 2018, the GoM increased the education budget by EUR2m, or more than 11 percent (EUR19.4m in total), to offset the loss of revenues mainly from fee-based tuitions. In 2019, the higher education budget was maintained at the level of EUR19,4 m, as this amount was found by the GoM to be sufficient to cover all the costs imposed by the new HE funding reform. Moreover, a fee for student’s poor performance has been applied (vs. the unconditional long-term enrolment of poor performers in the past) becoming a source of UCG income. B. Efficiency promoted by the implementation of NQAF The establishment of the NQAF resulted in the following: 10Versus fee-based student enrolment model in the past that tended to stimulate over-enrolment in order to complement the government contribution Page 46 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) (1) Efficiency measures were introduced at HEIs that resulted in a reduction of the number of HEIs from 13 to 9 HEIs and a decrease of academic programs from 300+ to 208. At the same time, the public resources have been channeled to the most relevant courses from the market perspective, thus promoting efficiency and strategic direction of HEIs development. (2) Mandatory 25 percent of practical education at HEIs was applied as part of each study program. This reform promotes development of skills that are relevant for the market, thereby positively affecting students’ employability, leading to Montenegro's economic growth in the future and stronger personal, economic, and social returns from higher education. (3) The external evaluations of all HEIs helped build capacity within MoE and HEIs in the area of systematic self-evaluation and evidence-based decision-making that in the future will lead to more efficient decisions. (4) A new study model (composed of 3 years of Bachelor + 2years of Master + 3 years of PhD studies) was implemented and is aligned with the prevailing models in the EHEA and the Bologna process. Harmonization will enable international integration, facilitate mobility programs (for students, teachers, researchers, and technical and administrative staff), promote cooperation between universities. The new study model will likely positively affect the Montenegrin education system's efficiency, quality, and its overall competitiveness. (5) A comprehensive tracer study was funded that built capacity and provided tools to MoE and HEIs for setting up monitoring and evaluation mechanisms on transition and skills mismatch issues between the HEIs and the employers. MoE intends to repeat the study as well as to apply its methodology in assessing the relevance of vocational and secondary education. C. Efficiency promoted by the Scholarship Program for Master’s, PhD, and Postdoctoral Studies The scholarship program assessment report11 provides valuable short-term insights on the program results related to its economic impact, such as: (1) the employment of MSc and PhD scholarship has increased by 104 percent; (2) all postdoc scholarship holders progressed in their career (promotion of at least one level), in that, the majority become leading Montenegrin researchers in their scientific fields; (3) the program had positive impact on participation in international projects given that a number of researchers have connected with researchers in other institutions and launched international projects; (4) the presence of Montenegro in international associations and organizations has increased (it relates to both, personal membership or Montenegro’s membership in international organizations). In addition, given that the development of the Montenegrin scholarship program was based on EU Marie Skłodowska-Curie actions (MSCA)12 scheme as well as it covered areas targeted by the Erasmus (+) program, both programs can serve as proxy for the expected results in long-term perspective. o The MSCA ex-post evaluation under FP7 and the interim assessment under the EU Horizon 2020 (H2020), indicated the following: (1) The MSCA was effective in boosting the career of 11 “Higher Education and Research for Innovation and Competitiveness” (HERIC) project. Technical Assistance for Evaluation of National Excellence Scholarship Programme implemented through HERIC project Ref. No. MNE-HERIC-81180-P122785-IC-CS-19- 2.2.2.3 Report on Evaluation of National Excellence Scholarship Programme by dr Miroslav Trajanović 12 The Marie Skłodowska-Curie actions (MSCA) provide grants for all stages of researchers' careers - be they doctoral candidates or highly experienced researchers - and encourage transnational, intersectoral and interdisciplinary mobility. The MSCA enable research-focused organisations (universities, research centres, and companies) to host talented foreign researchers and to create strategic partnerships with leading institutions worldwide. Additional details available at: https://ec.europa.eu/programmes/horizon2020/en/h2020-section/marie-sklodowska-curie-actions Page 47 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) researchers. Around 60 percent of past MSCA fellows believe that it would have taken them more time to attain their subsequent career stage without the MSCA fellowship, and 12 percent believe they would not have attained the subsequent career stage at all; (2) The program helped produce the next generation of leading researchers (i.e., MSCA fellows are twice as likely as the average researcher to have publications that belong to the Top 1 percent, Top 5 percent and Top 10 percent of cited publications); (3) MSCA had a key role in "building competence in the long term, focusing strongly on the next generation of science, systems and researchers, and providing support for emerging talent; (4) Mobility was the key tool to develop international cooperation; (5) MSCA contributed to cross-sectoral mobility of researchers during and after the MSCA projects (11 percent of MSCA fellows mainly hosted in the academic sector during their fellowship moved to the non-academic sector after the fellowship). o Impact evaluation (2019) of the Erasmus (+) 1314program revealed the following findings relevant for the HERIC15 scholarship program: The program’s graduates: (1) developed employability skills ( e.g., enhanced communication skills, ability to adapt sector- and field-specific knowledge in new situations); (2) acquired technical, inter-personal and inter-cultural skills and competences, self- confidence, ability to achieve goals, and social and cultural openness; (3) 40 percent were offered jobs with the companies/ organizations in which they did their traineeship while around 10 percent started their own company, with many more (75 percent) planning to do so in the future; (4) around 80 percent found their first job after graduation in less than three months; (5) embraced new learning methods. While the higher education staff who took part in an Erasmus+ Mobility is (1) more innovative and in touch with the labor market compared to the peers; (2) has impact on international competitiveness of the home HEIs; (3) has impact on the use of new teaching methods, development of curricula and teaching activities. Moreover, o The project established an infrastructure that facilitates access to international funding for education. Explicitly, the MoE set up an office to support institutions with accessing bilateral and multilateral programs for students and faculty (including enhancing participation in the EU Erasmus+ Program). Further, HERIC helped build capacity of the new office to effectively and efficiently promote mobility proposals across 80 international universities. Between 2014 and 2018, there were 1,200 mobilities of staff and students from Montenegro and about 450 staff and students coming to Montenegro; both numbers continue to grow. o Efforts under HERIC informed the decision to support curricular reform with a mandatory 25 percent of practical education at HEIs, to be applied as part of each study program. This reform promotes development of skills that are relevant for the market, thereby positively affecting students’ employability, leading to Montenegro's economic growth in the future and stronger personal, economic, and social returns from higher education. o HERIC helped establish a systematic cooperation with the Montenegrin Scientific Diaspora promoting efficiencies by accessing or increasing chances for participation in international R&D 13 Erasmus+ supports traineeships (work placements, internships, etc.) abroad for students currently enrolled in higher education institutions in Programme countries at Bachelor and Master level as well as for doctoral candidates. These opportunities are also open to recent graduates. 14 European Commission (2019). Erasmus+ Higher Education Impact Study. 15 Between 2014 and 2018, two million students and staff in higher education undertook a learning, training or teaching period abroad with the new Erasmus+ programme. The objective of the study was to assess the impact of Erasmus+ mobilities on staff, students and higher education institutions. It was based on almost 77,000 survey responses from these groups. Page 48 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) projects. This increases Montenegro’s international connectivity, application of global best practice as well as enhancing changes for participation in international R&D. HERIC helped develop a Diaspora database, that is being kept up-to-date and actively promoted to attract more scientists residing abroad. As a result of all these efforts, MoS currently systematically encourages the Diaspora to participate in national and international R&D projects. D. Economic value of the implementation of CoE and the Collaborative R&D Subprojects Establishment of Pilot CoE and funding of eight Collaborative R&D Subprojects (grants) set a stage for an entirely new approach to R&D funding in Montenegro towards multidisciplinarity, research quality, market applicability, sustainability, connectivity to international R&D networks and execution of projects in partnerships. All these elements have a critical meaning for developing innovation and thus future economic growth of the country. o The program has created new capacities across Montenegrin HEIs to tap into international funding programs. Both, CoE and Collaborative R&D Subprojects encouraged cooperation between domestic and leading international institutions as well as business sector. This resulted in creation of high-quality and sustainable partnerships that show great potential for integrating research into mainstream economic drivers of Montenegro. Already some of the winning research teams were able to secure external financing as a follow up activity. Until October 2019, the team leading the project LOVCEN became a partner in six EU grants, including from Horizon 2020, also teams of the KATUN and PRODE projects were successful in receiving competitive funding. o All projects produced direct or indirect impact on economy. Out of 22 products or services developed via CoE and Collaborative R&D grants, 15 were applied in the public or private sector. Among sound examples is the project HLA-MNE that created the first HLA (human leukocyte antigen) typing and HLA laboratory in Montenegro. The laboratory contributed to a significant cost reduction for the national health related budget, improving the overall quality of healthcare (before the project, all HLA typing was performed in Croatia imposing among others high transportation costs). Another example of economic impact is creation of a start-up company under the project TELEMONTEKG in the field of rapid diagnosis of disorders in heart rhythm. o The CoE and Collaborative R&D programs imposed the requirement of contribution (either financial or in kind) by the participating partners. This condition has been transformed into the obligatory financial contribution by the grant holders in all new programs managed by the MoS (e.g. creation of new Centers of Excellence, grants for R&D and innovation). Such approach supports economic efficiency, additionality and is aligned with international best practice. o The design of Montenegro’s CoE and Collaborative R&D Subprojects was aligned with the overall direction of the EU Horizon 2020 supporting both research an innovation. Montenegro followed the practice by the developed or MIC economies who had applied the CoE approach, including e.g., Austria, Denmark, Croatia, Estonia, Finland, Germany, Ireland, Japan, Korea, Netherlands, Norway, Portugal, USA, Sweden, and many more. Evaluations of CoE initiatives have indicated positive results across economies16 which can be relevant also in the case of Montenegro, e.g. in Estonia program helped improve the competitiveness of Estonian R&D, as shown by growing success in competitive applications to the EU’s research funding programs; in Finland, the scheme promoted the development of priority areas in HEIs; in Germany research output of program 16 OECD (2014). “Promoting Research Excellence: New Approaches to Funding”, OECD Publishing, Paris. Page 49 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) increased significantly in quantitative and qualitative terms; in Ireland, the scheme improved the reputation of the Irish research base and increased Ireland’s attractiveness as a location for industrial R&D; in Korea, early-stage research was fostered, research quality improved, and better linkages with industry were established; in Norway, the CoE scheme was particularly successful in terms of promoting researcher recruitment, strengthening the internationalization of Norwegian research and creating networks through national and interdisciplinary collaboration. o It shall be noted that the evaluation studies from other countries do not provide information on the program’s return of the investment since it is difficult to measure results of initiatives with a multidimensional character (such as institutional reforms, cooperation with global partners, development of linkages with the private sector, development of the intellectual property framework and commercialization processes at universities). Yet, some researches investigating the role of CoEs argue that CoEs have the ability to revitalize the academic system by providing platforms for university-industry and triple-helix relations and promote bottom-up priority setting among scholars and universities and attract talent (Langfeldt et al. 201517; Borlaug 201618). o Given the above experience from other economies that applied the CoE approach, it can be assumed that CoE and Collaborative R&D Subprojects may bring tangible results in the long-term perspective. o Under HERIC, two reports mapping the existing research equipment in Montenegro were conducted in 2013 and 2019 to avoid procuring the same capital equipment and optimize its use through equipment sharing among the HEIs. As next steps, HEIs plan to develop procedures for the optimal use of the equipment and open access. Main factors that may affect the HERIC results. A couple of factors can be identified that potentially can affect the positive results generated by the HERIC. These are: (1) UCG has a relatively low budget for R&D and to a large extent has built its strategy on tapping into external funds, which is risky. Competitive grants model that HERIC has initiated is a welcome development yet not sufficient. Most of the researchers do not have experience hence skills in writing grant proposals. Among solutions could be to provide certain level of basic R&D funding (which is not competitive) to help researchers to perform research to generate preliminary data necessary for future grant proposals. (2) The efforts MoE is making to promote strong research practices in higher education, including through career development at the university, could be strengthened. Therefore, it would be strongly recommended that the two ministries (MoE and MoS) agree on a joint strategy enhancing both teaching and research activities. 17 Langfeldt, L., Benner, M., Sivertsen, G. et al. (2015) ‘Excellence and Growth Dynamics: A Comparative Study of the Matthew Effect’, Science and Public Policy, 42/5: 661–75. 18 Borlaug, S. B. (2016) ‘Moral Hazard and Adverse Selection in Research Funding: Centers of Excellence in Norway and Sweden’, Science and Public Policy, 43/3: 352–62. Page 50 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Page 51 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) 6.1. Borrower’s ICR IMPLEMENTATION COMPLETION AND RESULTS REPORT Higher Education Research for Innovation and Competitiveness – HERIC Project (P122785) This report is prepared by the HERIC project management team, July 2019 The HERIC project development objective was to strengthen the quality and relevance of higher education and research in Montenegro through reforming the higher education finance and quality assurance systems and by strengthening research and development capabilities. The Project activities were carried out over a period of seven years, from May 2012 until the end of June 2019, with total Project amount of €12 million ($16 million). The Montenegro HERIC Project was placed within the overarching framework of the Government of Montenegro’s (GoM) strategies for European integration in the areas of science and research aiming to help the Montenegrin authorities deepen ongoing reforms and align them with the broader EU integration agenda. The HERIC Project’s main beneficiaries were students, graduates and academic staff, who benefited from implementing supports for quality assurance and finance reform in higher education, as well as internationalization efforts for human capital capacity building. Universities and research institutions also benefited from financing higher education reforms and were supported by grants financed by the Project in order to incentivize participation in higher education reforms and support larger, more impactful R&D subprojects which contributed to development of a more competitive research environment. Business sector representatives were involved as indirect beneficiaries of Project activities implementation, and GoM institutions were supported through capacity building activities financed by the Project. The Project comprised 4 components: − Higher Education Finance Reforms and Implementation of Quality Assurance Norms − Human Capital Development through Internationalization Initiatives − Establishing a Competitive Research Environment − Project Management and Monitoring and Evaluation Description of the project results through Components Page 52 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Component 1 - Higher Education Finance Reforms and Implementation of Quality Assurance Norms The HERIC Project Component 1 was designed to help the GoM implement key higher education reforms, as defined in the Strategy on the Development and Financing of Higher Education for the period 2011-2020, which was later revised in 2016. This component supports the financial and technical mechanisms to achieve the effective implementation of these complex strategic goals. This component financed technical assistance, trainings for capacity building, and grants to faculties to support the strategic reforms implementation. Subcomponent 1.1 - Higher Education Finance Reforms The HE finance reform was crucial for modernization of Montenegrin higher education system, as the analyses of the previous financing model pointed to its serious deficiencies. The system was in need of a sustainable funding strategy that would build on global good practices, in order to design and put in place an effective resource allocation model adapted to the reality of Montenegro and address the specific challenges faced by its HEIs, promoting flexibility and innovation throughout the higher education system. This reform process involved the preparation of a comprehensive analysis of the overall performance of Montenegrin higher education funding system, with recommendations on how to improve it, the proposal of a new funding model aimed at achieving financial sustainability, and a detailed roadmap to guide its implementation. Out of the two suggested scenarios for achieving that purpose, the GoM found the contractual model as the most suitable for Montenegrin circumstances and started working on this solution. The Performance Contract between the GoM and University of Montenegro (UoM) was prepared, in line with the recommendations, designating detailed criteria, standards and performance indicators in the funding scheme and obligations of both GoM and UoM, and ultimately signed. It also provides for mechanisms of additional, complementary funding that would lead to greater sustainability. The new, contractual, funding model of public HEIs was prescribed by the Law on Higher Education (adopted in 2017) and effective from academic year 2017/18. It focuses largely on monitoring the efficiency and effectiveness of public allocations aimed at national priorities, improves access to higher education, encourages education and research through which future economic growth will be improved. The new model also envisages free studies for students of the first and second cycles, i.e. undergraduate studies starting from 2017/2018 academic year, and master studies starting from 2020/2021 academic year, so as to provide equal opportunities for all students of public HEIs, regardless of their financial background. This involved a significant increase in the budget allocations for higher education. In 2018 the budget for UoM has been increased Page 53 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) by 2 million EUR (€19.4 million in total). At the moment, the call for enrolment of third generation of first cycle students, according to the new funding model, is underway (academic year 2019/20) – each year the call envisages about 3500 students. Subcomponent 1.2 - Higher Education Quality Assurance and Relevance Activities National Quality Assurance Framework in Montenegro This subcomponent supported the improvement of administrative capacities of the Council of Higher Education, the MoE’s Higher Education Department and other key actors in managing quality assurance activities. The project funded technical assistance for meeting the requirements for membership in the European Association for Quality Assurance (ENQA) and the European Quality Assurance Register (EQAR) with the ultimate goal of establishing an internationally recognized QA body from Montenegro. Through this process, a set of QA documents was prepared and adopted - including guidelines, procedures, forms etc. - with the aim of making the overall QA process more relevant and transparent, putting a central focus on their full compliance with Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), which play an important role in the development of national and institutional QA systems across the European Higher Education Area (EHEA). Following the EU guidelines and general European practice, a new and independent Agency for the Control and Quality Assurance of Higher Education was established by the GoM in November 2017. The Agency is responsible for accreditation and re- accreditation processes, i.e. external evaluation of HEIs and it took over the procedures from the Council for Higher Education, right in time to conduct the new accreditation of HEIs according to the new model of studies, as prescribed by the 2017 Law on Higher Education. It also took over the set of QA documents, previously prepared through HERIC for the Council, and updated them in line with the new ESG adopted in 2015. HERIC continued providing support to the new QA body through capacity building of its staff, this being one of the key steps at the beginning of building a solid foundation for the future work. The exchange of ideas and expertise, promotion of communication and cooperation with agencies and institutions in other countries have greatly contributed to building effective mechanisms within the national system and providing guidelines for their work in accordance with best practice. The Agency’s commitment to the objective of becoming a reputable body delivering on the core values of QA, as enunciated in the ESG, has already been confirmed by the fact that in July 2019, they became an affiliate member of ENQA, which is the first and important step towards full membership in this association. External Evaluation of HEIs in Montenegro As part of the quality assurance initiatives, to provide Montenegrin HEIs with institutional Page 54 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) evaluation experiences that follow the best European practice, two cycles of coordinated external evaluations of all Montenegrin HEIs were carried out, i.e. initial and follow-up evaluation. The entire process was completed by an independent service of the European University Association (EUA), i.e. the Institutional Evaluation Program (IEP), which is listed on the European Quality Assurance Register for Higher Education (EQAR) and is a full member of the European Association for Quality Assurance in Higher Education (ENQA). The initial evaluation completed in 2013/14, resulted in individual evaluation reports for each institution and a system review report which highlighted the shared issues and challenges, facilitating a dialogue among all key actors and stakeholders, including the government authorities. The reports provided a set of recommendations for improving the identified deficiencies, both at institutional and at national level, which served as valuable input for further development and improvement of institutions, as well as the overall system. This particular process has significantly contributed to the improvement of the institutions’ management and governance mechanisms, as well as to the development of new quality assurance procedures. The measures taken on a system level can clearly be seen in the strategic and legislative documents adopted in the years that followed the evaluation. Crucial changes related to: • the introduction of a new study model (3y Bachelor + 2y Master + 3y PhD studies), which the dominant model in EHEA; • introduction of a new funding model, i.e. the contractual model; • free undergraduate studies at public HEIs starting from 2017/2018 academic year; • free master studies according to the reformed model of studies starting from 2020/2021 academic year; • introduction of mandatory 25% of practical education at HEIs, as part of each study programme; • establishing an independent Agency for control and quality assurance of higher education. It is exactly the success of implementation of the recommendations from the initial evaluation that was the subject of the follow-up evaluation of HEIs carried out in 2017/18. The new, follow-up, reports highlighted the progress between the two evaluations and how the systemic weaknesses were addressed, and provided additional recommendations for further improvement. These reports continue to be a valuable foundation for further development, both for the MoE as a body responsible for the entire HE system, as well as for HEIs themselves in the development of their own policies. The entire process also provided all participants, particularly HEIs, with valuable practice in selfevaluation procedures in line with the best European practice. Page 55 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Tracer Study In order to generate objective, standardized information on experiences of graduates upon completion of their studies and use this data in effective policymaking and interventions, a Tracer Study on higher education graduates was implemented. The process envisaged the preparation of feasibility study that provided a detailed analysis of similar activities in Europe, including extensive recommendations about implementing the most adequate and sustainable model/concept of tracer study mechanism adapted to Montenegrin circumstances. This was followed by the procurement of hardware and development of tailor-made software for Tracer Study implementation, all in line with the findings and recommendations of Feasibility Study. The first survey, conducted within the project, covered students who graduated from Montenegrin HEIs in 2009 and 2013. It focused particularly on information about whether the graduates managed to find jobs, including how long took them to do so, types of jobs, starting salaries, career progression, as well as the relevance of formal education. The subsequently prepared analytical report represents a highly relevant material and a new source of information useful for various stakeholders, including policy makers, high school students and university students, university managers, employers, etc. In addition, this analysis can also serve as an initiative for future development of data collection system on education and to further promote the regular implementation of research activities on young graduates at certain time intervals. This entire process, therefore, resulted in the establishment of an entirely new framework for future tracer studies. Additional activities in the field of data collection for policy-making purposes, also involved considering the perception of employers. Namely, the project Component 4 involved conducting two surveys of employers’ perception of relevance of HE programmes. The first one was done in 2013, and the second one towards the end of the project, in 2019. The surveys showed that the overall context for assessing higher education in Montenegro has been improving, with the sector seen by employers in high regard. The information gathered through this survey is extremely useful for future development of HE, as it gives a clear overview of labour market needs in terms of the relevant human resource education. Plagiarism prevention framework In order to strengthen the quality of academic environment a new framework for plagiarism prevention has been developed. In 2016, a Feasibility study on the proposed tailor-made system(s) for the prevention of plagiarism in Montenegro was prepared, based on EU practices, after which all HEIs in Montenegro have been provided with a two-year access to one of the best plagiarism detection systems – iThenticate. As a further step in this field, the GoM adopted the Law on Academic Integrity in March 2019, making Montenegro the first country in the region, and beyond, which regulates the issues of academic integrity by a special law. The primary objective of this law is Page 56 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) further improvement of higher education quality by emphasizing the importance of transparency, promoting academic and democratic values in higher education and developing an overall environment in which respecting ethical principles is one of the key missions of each institution. It also encouraged a number of institutions to take further actions in this particular regard within their own strategic documents (e.g. Strategy for Development of University of Montenegro 2019-2014) and in general increased their engagement through training and coordination of events aimed at raising awareness on this matter. Development of National Qualifications Framework The project design envisaged financing the development of a National Qualifications Framework (NQF) for all academic subjects taught in the higher education sector. This activity was taken over by a separate entity during implementation, utilizing EU funds. Hence, although the project did not support the activity financially, the concepts and the implementation support for this activity was complementary to the structure and outcome HERIC goals. Subcomponent 1.3 - Competitively Awarded Grants to Incentivize Participation and Support Capacity Building After the first external institutional evaluation was completed, MoE developed and implemented the Institutional Improvement Grant scheme for HEIs. The overall objective of this grant scheme was to strengthen the administrative and operational capacities of HEIs so that they are in better position to: take part in the development of quality assurance and finance reforms, including the provision of data, participation in meetings and data collection; contribute to the refining of strategic findings and documentation; and implement the recommended strategic reforms at HEIs and in specific programs. The scheme implied awarding grants of up to 50,000 EUR for one- year projects aimed at institutional improvement, with minimum co-financing of 10% of the requested grant amount. The designed institutional improvement grant scheme was successfully implemented within the two calls published in 2016. In total, 16 grants were awarded, with total value of projects amounting to €778,957. The projects’ objective s spanned from curricular improvements and development of strategies, to procurement and modernization of IT and laboratory equipment, etc. All projects were successfully implemented meeting the overall grant scheme objectives. An added value to this process is the fact that the project teams got valuable experience in project management and implementation of procedures following the WB guidelines. It should be noted that the project design anticipated that 20 faculty and institution subprojects would be funded (with grants of €35,000 in average). However, due to larger contracted grant amounts (€43,419 on average), 16 projects were successfully implemented. Page 57 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) No. of institution Grant Co- Total Average grant per subprojects amount financing amount project 16 €694,703 €84,255 €778,958 €43,419 Component 2 - Human Capital Development through Internationalization Initiatives Subcomponent 2.2 - Promoting a Scholarship Program for Master's, PhD and Postdoctoral Studies Scholarship Program for Master’s, PhD and Postdoctoral Studies This subcomponent financed the needed technical assistance for designing a targeted scholarship scheme for the national interest areas of study, and provided scholarships to master’s, doctoral and postdoctoral students, according to the developed scheme. This particular subcomponent was jointly implemented by the MoE and the MoS. The envisioned scholarship scheme was developed using the best international EU experience/practices such as Horizon 2020 – Marie Skłodowska-Curie Scholarship Program and was adapted to the Montenegrin context. Two calls were published, in 2015 and 2016, each involving a detailed communication campaign, including series of press articles, info days, promotional material, etc. to encourage better participation and strongly underline the scheme’s objectives. The objectives of the National Excellence Scholarship Programme were to: - Strengthen national capacities for research, innovation and competitiveness; - Transfer and apply the knowledge acquired through academic development and career development of researchers in Montenegro; - Support internationalization initiatives; and - Strengthen economic sector by investing in human resources. The Programme for Master’s Scholarships was implemented by the MoE, whereas the programme for PhD and Postdoctoral Scholarships was implemented by the MoS. Through this Programme postgraduate students and young researchers had the opportunity to study and conduct research at prestigious universities/research institutions abroad (officially ranked among top 500 in the world), in order to gain international research experience and transfer the knowledge into the national science and innovation system. Below is the overall statistics of the programme. Page 58 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Statistic for 2015 and 2016 MSc PhD Postdoc Total Total number of applications 190 75 18 283 Number of evaluated 147 68 18 233 applications Number of contracts signed 86 32 14 132 Number of scholarships 84 31 13 128 completed Contracted gross amount (€) 644,972.51 363,909.00 205,474.00 1,214,355.51 The evaluation of the National Scholarship Programme was conducted after the implementation of scholarships in order to assess the impact and effectiveness of the programme in relation to the defined objectives. The resulting report and analysis clearly indicated that all program objectives were successfully fulfilled underlying that the structure and implementation of the program were evaluated as excellent by its beneficiaries. It also provided additional recommendations which will be considered by the MoE and MoS for improving future similar programmes. Following the model of the HERIC Scholarship Program, a new PhD scholarship program was designed and launched by the MoS in 2018, providing scholarship opportunities of up to €50,000 for a 3-year period. Through this program, the MoS encourages candidates to enroll Montenegrin universities and provides them with research opportunities and study visits abroad. The goal here is to reach the number of about 50 young researchers (PhD students) in 3 years period. For the 2018 Call, 19 PhD students were awarded and the second Call launched for 2019 is currently in the evaluation phase. Several HERIC scholarship beneficiaries (MSc scholars) used this opportunity to continue the next stage of their education in Montenegro. Subcomponent 2.3 – Technical Assistance to Support Internationalization Initiatives Study programmes in English language In order to attract more international students to Montenegrin HEIs and enhance Montenegrin higher education system competitiveness a Feasibility Study on Study Programs in English was prepared. It identified study programs with strong and clear potential to be operational in English language, based on the assessment of the present capacities, which resulted in the proposal to continue internationalization process in two Page 59 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) stages. The first stage included the development and implementation of four existing Master level study programs in English, identified as the most appropriate and adequately equipped for this. The second stage implied designing new study programs in English to be initiated in the future, according to strategic development priorities. Improving the foreign credential recognition process MoE has undertaken the initiatives to improve the process of foreign HE diploma recognition in the country, which is one of the essential elements for facilitating international mobility. To this end, HERIC provided necessary professional assistance in preparation of a policy paper that would serve as a starting point for upgrading the existing recognition process, focusing particularly on quality assurance in this field. The improved procedures will ensure that the recognition policies are based on the Lisbon Recognition Convention, a fundamental document regulating the international academic recognition in the European region, and provide suitable instruments to assist credential evaluators in practising fair recognition and observing all of the legal obligations. The MoE established working group which is currently working on this matter, using inputs from this document. Scientific Diaspora Montenegro has enormous intellectual potential in the Diaspora, and its scientists with established positions at recognized international institutions can significantly contribute to the development processes in Montenegro. HERIC supported this process by providing technical assistance in preparation of the Analytical study on the Montenegrin scientific Diaspora, which was adopted by the Government in November 2014. The Study provided the MoS with preliminary data about scientists abroad which was later integrated in “Science Network” – Montenegrin research information system. This database is constantly updated and researchers are continuously invited and encouraged to join and create their profiles in it. The Study also provided guidelines and recommendations for better networking and cooperation with Diaspora. In November 2017, in cooperation with the Montenegrin Chamber of Economy, MoS organised the Conference on Economy “Montenegro 2017”. The event was attended by around 500 participants from government, business and academic communities, as well as by 35 recognized Diaspora scientists which were supported by HERIC funds to attend this event. The aim of the Conference was to network experts from the research and business community, as well as to strengthen links between our Diaspora scientists and local experts representing academic and business community. Diaspora scientists presented themselves through panel discussions with renowned local and foreign speakers and expressed their readiness for cooperation and active participation in policy Page 60 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) development and R&D activities in Montenegro. The MoS continuously encourages participation of Diaspora members in a number of initiatives, by involving them in evaluation processes as well as by fostering cooperation with them in national and international R&D and innovation projects. Component 3 – Establishing a Competitive Research Environment The objective of this component was to provide the relevant support needed to transform higher education and research into the mainstream economic drivers for Montenegro. These efforts are also directed towards better connections with EU in the areas of science and research, and enterprise and industrial policy. To this end, the focus was on: • strengthening links between HEIs, R&D centres and firms at national and international level; • strengthening the human capital and research infrastructure capacities by promoting accessibility and mobility of scientists; • modernizing laboratory equipment for the needs of high quality scientific and applied R&D; • stimulating technology commercialization and patenting activities. The main activities included: • establishment of the first Pilot Centre of Excellence (€3.4 million); • implementation of Collaborative Research and Development Subprojects (€2.6 million). Subcomponent 3.1 - Establishment of a Pilot Centre of Excellence Establishing the CoE was defined as a core component of the Government’s science strategy, as articulated in the Law on Scientific Research Activity. The initial step in this process was to design the new CoE program by preparing a set of guidelines and procedures for its implementation process. The criteria for the competitive selection process of the pilot CoE was defined in the designed CoE Manual. On May 31, 2013, the MoS launched the Call for Proposals and the application process ended on September 20, 2013 with ten (10) applications received. After the administrative check, evaluation process was organized in two stages. In first stage, each proposal was reviewed by two international experts-peer reviewers in their specialized scientific area, focusing particularly on research aspects of the proposal. The second Stage was carried out by the Final Evaluation Committee (FEC) which reviewed the proposals, carried out a joint meeting and interviews with applicants. FEC largely focused on innovativeness, competitiveness, sustainability and commercial aspect of the project proposals. The grant was ultimately awarded to the University of Montenegro (UoM) - Faculty of Page 61 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Electrical Engineering for implementation of the project “Center of Excellence in bioinformatics” (BIO-ICT). The project started on June 1, 2014 and lasted until March 31, 2019. Total awarded grant was €3.418 million. The aim of BIO-ICT was to boost the application and use of the latest ICT technologies in the fields of sustainable agriculture, monitoring of crops, forest and water/sea ecosystem, development of techniques for controlling and reducing air pollution, analysis and standardization of food products, control of land quality, and improvement in the public health area. Several project objectives were set to meet the overall identified needs: − development of new scientific approaches and methods for improving products and services in life sciences (primarily agriculture, health and ecology); − strengthening interdisciplinary research (4 national research institutions involved); − strengthening cooperation with international research entities (2 international partners); − establishing cooperation with private sector in Montenegro (2 domestic companies), and − improving potential for creating commercial innovations through the commercialization of knowledge and IP. In June 2016, two international experts carried out mid-term evaluation of the project implementation (Prof. Andrea Corradini from Denmark and Dr Bagrat Yengibaryan from Armenia). They treated many aspects of the project including: quality of technical activities and ideas, commercial potential, management of project activities and the project as a whole, legal and organizational framework of the CoE, plans for sustainability and dissemination of the project, etc. Their reports included both comments and recommendations which were carefully considered for implementation of further activities. In October 2017, German consulting company Detecon was engaged to develop a business plan and sustainability study. It proposed establishing the CoE and a separate entity within the University of Montenegro as the most adequate form of organization of the BIO-ICT project, particularly in terms of its sustainability. Accordingly, a special organizational unit of UoM called “Institute – Center of Excellence for Research and Innovation” was established by the GoM’s decision of 10 May 2018. It has taken over the existing resources from the BIO-ICT project, as well as its basic research concept, which will be expanded to include all researchers and research groups from the UoM so as to create synergy and stimulate networking between different disciplines. Amendments to the UoM’s Statute were successfully approved and in March 2019 the Act on Systematization was adopted. By the end of March 2019, infrastructure improvement was mainly focused on creating a new co-working space at UoM serving as a new, modern hub for researchers and PhD students. The final step in this entire process was to measure the impact of CoE project implementation, as well as to develop the framework for future CoEs, based on entire implementation experience. This assessment was conducted by the Diaspora expert (Dr Page 62 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Andrej Mihailovic) and completed in June 2019. BIO-ICT was the first and the biggest scientific multidisciplinary project funded by the GoM, where researchers from different fields and institutions - namely Faculty of Electrical Engineering, Biotechnical faculty, Institute for Marine Biology and Institute for Public Health, cooperated on a daily basis and conducted different multidisciplinary analyses and experiments. Some of the results achieved through the project include: • 4 new laboratories were formed and fully equipped, and 10 of them were upgraded. A total of € 1.24 million was invested in modern equipment. • 90 researchers were involved and trained in different stages of the project, and out of them 31 were young researchers (including 5 PhDs, 14 PhD students and 8 MSc students). A total of 19 people covered topics related to the BIO-ICT project in their master's and doctoral topics, 13 of them defending their thesis. • 5 patents have been implemented at national level, while the procedure for obtaining one international patent is underway. • Several innovative products and services have been developed: BIO-Portal, Live- Gate, SEMaR system, Smart Irrigation system, Smart spraying, Drone Photos Mapping, Sharp Eye. 5 of them have been applied in the public and private sectors. • Project staff has published 51 scientific papers related to the topics and activities of the project, 39 of which are on the SCI list. Furthermore, the GoM has set a goal in the current mandate to establish new CoEs that would be the drivers of scientific activity and the core of development and innovation activities in the country, in perspective areas for Montenegro. In September 2018, a new call for establishing new CoEs was announced, with end of March 2019 as application deadline. Funds amounting to 1 million EUR have been provided for a period of three years. 7 project proposals have been submitted, and all of them have entered the second stage of international evaluation which is currently underway. The evaluation and selection process are done according to the same methodology and rules designed within the HERIC CoE funding scheme. Subcomponent 3.2 – Supporting Collaborative Research and Development Subprojects To further strengthen R&D capacity in the national innovation system, this subcomponent supported the development and implementation of a competitive grants scheme - the Program for Collaborative R&D Grants (CRDS) with the main idea of supporting larger and more impactful R&D projects that foster international collaboration, generate commercial innovations and strengthen the Montenegrin economy. The grant scheme was implemented through two calls, the first one in 2013 and the second in 2014. In total, 47 project proposals were received, out of which 8 were selected and funded. Two-stage evaluation was applied for this scheme, as well as for the CoE call. The implementation of the 8 selected projects, worth €2.6 million, encouraged Page 63 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) cooperation between domestic/international institutions and business sector, which resulted in high-quality and sustainable partnerships that show great potential for integrating research into mainstream economic drivers of Montenegro, as well as for growing into the future CoEs. CRDS grant scheme supported both applied and development projects, within the following priority areas: Energy; ICT; Medicine and Health; Sustainable Development and Tourism; Agriculture and Food; and New Materials, Products and Services. The project design entailed subprojects between €150,000 and €400,000 with 12 subprojects funded (with average grant amount of €200,000). However, due to higher contracted grant amounts (€326,125 in average), 8 large projects with diverse partnership potential were contracted and implemented. Within these 8 projects, 20 public/private or international partnerships with domestic research institutions have been established. No. of CRDS Grant Average grant No. of projects amount per project partnerships 8 2,609,000 326,125 € 20 € The final step in this process was measuring the impact of CRDS projects implementation, which was conducted in June 2019. The impact of these multi- disciplinary partnerships is highly significant, among other things: • 7 new laboratories were established and equipped, and 4 existing ones upgraded. • a total of € 789,261.53 was spent on procurement of equipment. • 146 researchers were engaged, among whom 61 young researchers. A total of 32 individuals, covered topics related to the projects they participated in within their MSc and PhD theses, and 15 of them defended their theses. • 15 new products and/or services have been developed and most of them have already been applied in the public and private sectors (3 patents, out of which one international). • 317 scientific papers have been published, out of which 73 on the SCI list. Based on the HERIC CRDS grants experience, MoS has developed a new Innovation Projects Grant Program (2018-2020), and conducted the first Call for Proposals in 2018. Through this call 10 innovative projects were selected and approved for funding with the implementation period from one to two years. The total value of all 10 projects was €1.17 million. The MoS contributed €0.73 million (the amount of grants awarded was up to 100,000 EUR per project), while the rest was co-financed by the grant holders, largely by the private sector, as seven them were awarded to companies, and three to faculties. Each project was obliged to have a business partner in order to meet the requirements prescribed by the call. Furthermore, as the MoS intends to award grants for Innovative Projects each year, in July 2019 the second call for Innovative Projects. The call is open until 20 September, and the grants will be awarded to innovative projects that comply with the priority areas identified in the Montenegrin Smart Specialization Strategy which was adopted earlier Page 64 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) this year. Through the support of EU IPA funds, MoS has published another call for innovative grants with the aim to support the employment of PhDs in academic institutions and the economic sector, with a total budget of around €1 million. The evaluation stage for this call is underway. Through the CoE and CRDS implementation process, funded by HERIC, a lot of processes have been initiated and new funding instruments have been created. Moreover, research infrastructure has been significantly improved in the areas with the highest scientific and economic potential, quality and relevance of R&D outputs have been increased and cooperation with private sector and foreign research institutions strengthened. Subcomponent 3.3 – Technical Assistance to Strengthen the Design and Implementation of the Research Grant Program This subcomponent financed technical assistance and the related activities for managing and implementing the new CoE and grant programs and international peer review of the CoE and CRDS applications (already mentioned in the description of 3.1 and 3.2), as well as other supporting activities towards creating competitive research environment. The most relevant results are described below. Technical assistance on intellectual property and commercialization In 2017, CoE and the selected CRDS project teams were provided with support and advisory services regarding IPR and commercialization activities. This assignment was very helpful as the project teams had a limited understanding of market needs, difficulties organizing arrangements in an unfavorable legal and regulatory framework, and lack of knowledge particularly in technology transfer and science commercialization. The recommendations provided highlighted the need to carry out several priority actions for developing a better innovative ecosystem such as prioritizing key scientific areas for the future competitiveness of the country (smart specialization) and creating a technology transfer office to help with commercialization of knowledge developed by Montenegrin scientists. Also, TTO professionals should do the preliminary assessment of the commercial maturity of projects as well as to provide support to prepare various proposals (e.g. H2020). Than in parallel, universities need to develop their own IP legal frameworks and commercialization agenda and to support creation of technology-based start-ups spinning-out from research departments. Through HERIC, the MoS provided Technical Assistance for development of legislation for fiscal incentives (tax, customs etc.) for innovative activity within the business sector, in order to strengthen its competitive capacities. This purpose of this document was to analyse the legislative environment governing innovative activity in Montenegro, using the best practices and models of reform implementation in the region and EU, and Page 65 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) preparing draft regulations for proposed models of legal reforms, as well as fiscal incentives for R&D and innovation activities. Also, economic impact analysis of the proposed reforms and incentives was measured and analysed, and will be submitted to the relevant policy makers to further discuss the proposed legislative changes. Study on research equipment and creation of joint research facility Montenegro is the first country in the Western Balkans which conducted the process of research infrastructure mapping. It represents an instrument of identification and categorization of existing and planned research equipment at research and innovative institutions and organizations. Through the HERIC project, two detailed analyses were made for the preparation of the “Study on Capital and medium research equipment in Montenegro”, in 2013 and in 2019. The processes were carried out by developing specialized questionnaires and on-site visits to the institutions to obtain feedback for a comprehensive database on the existing and missing capital and medium research equipment. The resulting list of Capital and Medium Equipment were integrated in Montenegrin Research Infrastructure Roadmap document, adopted by the Government in 2015 and updated in 2019. This 2013 Study was actually designed in order to get an overview of the equipment in public ownership and avoid procuring the same capital equipment during the establishment of the CoE and competitive research grants. The updated, 2019 Study shows significant improvement and modernization of equipment and laboratories in Montenegro, especially in terms of new capital equipment at a number of institutions that had extremely high investment in the last 6 years. HERIC CoE, CRDS and HEIs grants greatly contributed to this improvement as around €2 million were invested in the equipment for new and existing laboratories. The 2019 Study also analyzed the current condition of the existing research equipment and the situation related to the equipment sharing, its more efficient and optimal use. All participants in the process expressed a general willingness to jointly implement projects, share equipment and resources. In the coming period it is envisaged that all the participants develop clearly defined procedures and protocols for the effective use of equipment and the overall open access. Also, a web portal with different services has been considered, focused on boosting and enabling sharing of resources and improvement of cooperation. Quantitative analysis for the Smart Specialization Strategy Through the HERIC project, several expert studies have been developed to support policy development. In 2018 a quantitative analysis “Mapping economic, innovation and scientific potential in Montenegro”, which was a key step in the process of development of the Smart Specialization Strategy development. The Strategy was was successfully prepared and adopted by the GoM in June 2019. Montenegro became the first country after EU28 countries with such important strategy Page 66 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) adopted. Smart specialization, as a key element of economic development policy, increases competitiveness of economy by focusing on a limited number of identified priority economic areas and linking research and innovation resources with them. Based on the comprehensive analyses, Montenegro has identified the following 3 vertical priority areas: sustainable agriculture and food value chain, sustainable and health tourism, energy and sustainable environment, with technological support of ICT as horizontal priority. Activities to support establishment of Science-Technology Park in Podgorica Science- Technology Park (STP) is an important part of the innovative ecosystem and this project has been identified and accepted as the GoM’s priority for its current mandate in the field of science and innovation. It is part of the Programme of Economic Reforms and recognized as one of the most important infrastructure development projects in Montenegro. By concluding the Agreement on the Establishment of the “Science and Technology Park of Montenegro” LLC between the GoM and UoM, adopted by the Government on January 17, 2019, new national infrastructure was established which will integrate innovative, scientific, entrepreneurial and economic capacities in Montenegro. The facility is located at the campus of the UoM, near the building of technical faculties and laboratories, and it is currently in the phase of rough construction works. HERIC project financed the preparation of the Terms of Reference for the new STP and the development of project documentation – Preliminary and Main Design of STP building, in order to bring this facility to a new purpose by renovating the existing building. It is planned that the construction works on the STP facility will be completed by the end of 2020. Moreover, HERIC supported STP with new IT equipment, website and visual identity development, and also provided funds to support the STP management staff in their capacity building and study visits. Component 4 – Project Management and Monitoring and Evaluation Subcomponent 4.1 – Project Management This subcomponent was designed to build capacity within the MoE and MoS to manage the day-to-day implementation of the HERIC Project, as well as monitor and evaluate its impact. It supported the establishment of a minimal Project Management Team (PMT) comprised of two Project Managers, two Assistants, one Grants Officer and one Environmental Expert. In addition, important elements in the implementation process were carried out by the Technical Service Unit (TSU) within the Ministry of Finance. The TSU provided fiduciary services (financial management, procurement, and disbursement related activities) to Page 67 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) HERIC, and does so for all other projects funded by the World Bank in Montenegro. In order to overcome bottlenecks due to a large number of grants and procurement processes initiated at the same time, an external Grant Procurement Specialist was engaged for a period of 6 months (Jan 2015 – July 2015). This expert contributed to prompt preparation of documentations and replies to CRDS and CoE grant beneficiaries, thus facilitating smooth and efficient running of procurement processes which was indispensable during that period. Subcomponent 4.2 – Project Monitoring and Evaluation This subcomponent provided funds for the financial audits of the HERIC Project and the audits of implementation of its grants subcomponents, as described in the Grants Operations Manuals (for the faculties, pilot CoE and research institutes). The project financial statements were audited annually by indep endent audit firm “BDO Belgrade” which is acceptable to the World Bank, according to their criteria. Clean opinion was issued by the auditors on project financial statements for the overall audited period and no particular issues were identified by the auditors. HERIC results promotional event The end of the HERIC project was marked by a showcase event held on 21 June 2019 in Podgorica. For this purpose, video material was produced and presented in very effective way, highlighting all the results as well as the impact the project has made in both achieving its development objectives and other results and outcomes that were reached beyond the project design. Page 68 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 6.2. Key World Bank’s disclosable ISRs http://documents.worldbank.org/curated/en/docsearch/projects/P122785 6.2. Borrower ICR • Implementation Completion and Results Report of the HERIC Project. Report prepared by the HERIC project management team. July 2019 6.3. Borrower’s semiannual progress reports on HERIC implementation • Semi-annual progress reports prepared by the HERIC management team. Reports for the period of time between July 2012 until December 2018 6.4. Bibliography on the evidence supporting assessment of the project outcomes 1. Accreditation/reaccreditation documents and procedures listed on the QA Agency’s website: http://akokvo.me/reakreditacije-pravila/ 2. Analytical report of data collected through the Tracer Study: http://www.heric.me/en/sadrzaj/analytical-report-data-collected-through-tracer-study 3. Advisory services on Intellectual Property and academic IP commercialization. HERIC project. Report prepared by Peter Lindholm. May 2017 4. Analytical study on the Montenegrin Scientific Diaspora (2014) 5. BIO-ICT project documentation (i.e. the pilot center of excellence) website: http://www.bio- ict.ac.me 6. Borlaug, S. B. (2016) ‘Moral Hazard and Adverse Selection in Research Funding: Centers of Excellence in Norway and Sweden’, Science and Public Policy, 43/3: 352–62. 7. CRDS projects websites/information: • http://www.cs-ict.ac.me/Partnership.php • http://prode.me/ • http://project-lovcen.me/page.php?id=4 • https://sust-marina.wixsite.com/marina • http://www.heric.me/en/grant-lams/grant-lams • http://www.katun.me/page.php?id=2 • http://www.heric.me/en/grant-telemont-ekg/grant-telemontekg • http://www.heric.me/en/kategorije-clanaka/projekat-hla-mne 8. Decision of the Government of Montenegro for UCG on establishing UCG CoE as a legal entity "Institute - Centre of Excellence for Research and Innovation": http://www.sluzbenilist.me/pregled-dokumenta-2/?id={920D939E-955D-4B6F-A8D0- 0FAAF5CEF453}# 9. Draft Performance Contract between GoM and UCG prepared by Radu Mircea Damian 10. Documentation on four UCG facilities adjusted for individuals with disabilities 11. Economic Conference with participation of diaspora: http://www.heric.me/en/news/economic- conference-montenegro-2017-meeting-montenegrin-scientific-diaspora 12. Evaluation of the National Excellence Scholarship Program. Report prepared by Miroslav Trajanović. June 2019 13. Evaluation reports on HEIs in Montenegro and a cross-cutting summary report prepared by the HERIC management team: www.heric.me/en/sadrzaj/evaluation-reports-higher-education- Page 69 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) institutions-and-cross-cutting-summary-report 14. European Commission (2019). Erasmus+ Higher Education Impact Study. 15. Final project completion reports for eight CRDS and one CoE 16. Framework for future CoEs and a competitive call for establishing a new CoE based on HERIC experience: http://www.mna.gov.me/rubrike/konkursi/196984/U-toku-konkurs-za- uspostavljanje-centara-izvrsnosti.html 17. Funding Mechanisms for the Development of Higher Education in Montenegro. Report Prepared for the Ministry of Education of Montenegro by Jamil Salmi. 2013 18. Impact Assessment for Completed HERIC Projects. Technical Assistance for impact assessment of implemented collaborative R&D grant projects. Report prepared by Danica Ramljak. June 2019 19. Implementation Completion and Results Report of the HERIC Project. Report prepared by the HERIC project management team. July 2019 20. Information about presentation of the Tracer Study to the stakeholders: http://www.heric.me/en/news/presented-analytical-report-data-collected-through-tracer-study 21. Information on the performance contract between the Government of Montenegro and UCG: www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUK Ewjd24nPxInjAhWOzIUKHYXBAvUQFjACegQIAhAC&url=http%3A%2F%2Fwww.gov.me%2FResou rceManager%2FFileDownload.aspx%3FrId%3D345248%26rType%3D2&usg=AOvVaw1QH5Lk5lX ONpCiwByKRu9J 22. Information on the new funding model described in the Law on Higher Education adopted in June 2017: http://www.mps.gov.me/ResourceManager/FileDownload.aspx?rid=286918&rType=2&file=Law %20on%20Higher%20Education.docx 23. Information on a new Office of the MoE responsible for bilateral and multilateral programs 24. Information about mobility opportunities published on the MoE’s website and www.europasscrnagora.me: http://www.mpin.gov.me/rubrike/medjunarodni-programi- mobilnosti-studenata; http://www.europasscrnagora.me/euroguidance/index.php 25. Information of the MoS’s follow-up program to the National Excellence Scholarship Program: • 1st Call published in 2018: http://www.mna.gov.me/rubrike/konkursi/187456/Javni- poziv-Ministarstva-nauke-za-stipendije-za-doktorska-istrazivanja.html • 2nd Call published in 2019: http://www.mna.gov.me/rubrike/konkursi/198773/Novi- Konkurs-za-dodjelu-atraktivnih-stipendija-za-doktorska-istrazivanja.html 26. Information on www.naucnamreza.me - a platform facilitating networking and cooperation between diaspora and scientists 27. Information on refurbished of student dorms: http://www.heric.me/en/news/contract- renovation-7-students-hostels-montenegro-was-signed; http://www.heric.me/en/node/957 28. Information on HERIC methodology applied to new MoS grants for innovation published by the Ministry of Science: http://www.mna.gov.me/en/ministry/189074/The-Ministry-of-Science- publishes-a-Call-for-awarding-grants-for-innovative-projects.html?alphabet=lat 29. Langfeldt, L., Benner, M., Sivertsen, G. et al. (2015) ‘Excellence and Growth Dynamics: A Comparative Study of the Matthew Effect’, Science and Public Policy, 42/5: 661–75. 30. Law on Academic Integrity, adopted in March 2019 31. Law on Higher Education, adopted in June 2017 32. Law on Innovation Activity adopted in 2016 Page 70 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) 33. List of accredited and licensed HEIs in Montenegro published on the website of the Ministry of Education: www.mps.gov.me/rubrike/enic-centar/153969/Akreditovane-i-licencirane-ustanove- viskokog-obrazovanja-i-studijskih-programa-u-Crnoj-Gori.html 34. List of 16 grants awarded in first and second call presented at HERIC’s website: • Grants awarded in 1st call: http://www.heric.me/en/news/call-proposals-institutional- improvement-grants; http://www.heric.me/en/sadrzaj/first-call-proposals-institutional- improvement-grants • Grants awarded in 2nd call: http://www.heric.me/en/sadrzaj/second-call-proposals- institutional-improvement-grants 35. OECD (2014). “Promoting Research Excellence: New Approaches to Funding”, OECD Publishing, Paris 36. Plagiarism Prevention Framework Feasibility Study prepared by Miroslav Trajanovic 37. Presentation and workshop on the use of iThenticate software (plagiarism prevention): http://www.heric.me/en/news/plagiarism-detection-software-be-used-all-institutions-higher- education 38. Public calls for admission of students for years 2017/18, 2018/19 and 2019/20 published by UCG: • A call for admission of 3486 students into the 1st year of studies for 2017/18: https://www.ucg.ac.me/objava/blog/10/objava/5267-upis-na-osnovne-studije-na-ucg- u-ponedjeljak-10-i-utorak-11-jula-2017-godine • A call for admission of 3501 students into the 1st year of studies for 2018/19: https://www.ucg.ac.me/skladiste/blog_10/objava_27029/fajlovi/Konkurs%20za%20upis %20studenata%20u%20prvu%20godinu%20osnovnih%20studija.pdf • A call for admission of 3486 students into the 1st year of studies for 2019/2018: https://www.ucg.ac.me/skladiste/blog_1240/objava_49072/fajlovi/02_165_4%20konku rs%20za%20upis%20studenata%20u%20prvu%20godinu%20osnovnih%20studija%20_1 _.pdf 39. Report on scientific equipment in Montenegro 40. Semi-annual progress reports prepared by the HERIC team on project implementation 41. Scholarship Management Handbooks for 2015 42. Scholarship Management Handbooks for 2016 43. Smart Specialization Strategy 2019-2024 44. Strategy for Development of Higher Education, 2016-2020 45. Strategy for Innovation Activity 2016-2020 with the Action plan 46. Strategy for Scientific Research Activity 2017-2021 47. The Revised Roadmap for research infrastructure in Montenegro for 2019-2020, Ministry of Science: http://www.mna.gov.me/ResourceManager/FileDownload.aspx?rid=365327&rType=2&file=Rev izija%20Mape%20puta%20za%20istra%C5%BEiva%C4%8Dku%20infrastrukturu%20Crne%20Gor e%20(2019-2020).pdf 48. The Marie Skłodowska-Curie actions (MSCA): https://ec.europa.eu/programmes/horizon2020/en/h2020-section/marie-sklodowska-curie- actions 49. Two calls for scholarships applications under the National Excellence Scholarship Program: • 1st call for scholarships applications: http://www.heric.me/sadrzaj/konkurs-za-dodjelu- Page 71 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) nacionalnih-stipendija-za-izvrsnost-2015 • 2nd call for scholarships applications: http://www.heric.me/en/sadrzaj/call-awarding- national-excellence-scholarships-cycle-ii-2016 6.5 Unemployment rates among the graduates with HE degree Number of % of unemployed Number of unemployed graduates graduates with HE unemployed graduates with HE degree* degree in total number with HE degree (age of unemployed persons group 25-36)* 2016 9500 19.8% 6000 2017 8500 19.4% 5900 2018 8000 18.8% 5700 Source: Ministry of Education *Calculated according to EUROSTAT methodology 6.6. Summary of the CRDS Assessment Report Some selected and notable results achieved under 8 CRDS projects include: (i) Clinical center in Podgorica established a new HLA (human leukocyte antigen) laboratory, contributing to a significant cost reduction for the national health related budget, improving overall quality of healthcare - with benefits for the patients in need of organ transplantations and genetic screening; (ii) AD Marina in Bar received the Gold Anchor certificate and a blue flag status, meeting strict environmental and safety criteria – with the ability to transfer knowledge to other ports and marinas in the country and wider region; and (iii) a monitoring station for lightning research developed putting Montenegro in the category of one of the three countries in the world with such system. This helps in preventing lightning damage of telecommunication, railway, electrical power and other technical systems. (iv) Development, validation and application of telemedicine systems TELEMONTEKG for the rapid diagnosis of heart diseases in Montenegro has enormous impact as heart diseases are among leading death causes in Montenegro. The new system was tested in 400 patients in Montenegro. In depth analysis of two CRDS projects leading in number of SCI publications (CS-ICT and LOVCEN) for the period between 2011-2018 reveals their significantly impact on total number of publications in Montenegro for that period. CS-ICT contributed 24.3% of total number of publications in computer science (signaling), while LOVCEN contributed 19.5% to total number of publications in agriculture (insect science). This is a great impact on countries scientific production and overall quality of science. Page 72 of 74 The World Bank Higher Education Research for Innovation and Competitiveness Project (P122785) Statistics on publications Number of publications (all scientific areas) Number of publications in computer 600 515 512 science (signaling) 500 453 408 384 80 341 63 400 293 239 60 300 31 200 40 22 15 16 13 13 12 100 20 0 0 2011 2012 2013 2014 2015 2016 2017 2018 2011 2012 2013 2014 2015 2016 2017 2018 Years Year Percentage of CS-ICT to total # of publications Percentage of LOVCEN to total # of publications in insect science (2011 - 2018) (2011 - 2018) Total CS-ICT Total LOVCEN Source: Danica Ramljak (June 2019). Impact Assessment for Completed HERIC Projects. Technical Assistance for impact assessment of implemented collaborative R&D grant projects. Page 73 of 74