World Bank Externally Funded Output on Evidence-Based Tertiary Education Policies for Better Employment in Belarus (P168224) International Experience in the Implementation of Learning Outcomes in Higher Education – Case Studies of Good Practice Minsk, 2018 Table of Contents EXECUTIVE SUMMARY 5 1 INTRODUCTION 6 2 SELECTED CASE STUDIES 8 2.1 UK – SCOTLAND 8 2.1.1 NATIONAL QUALIFICATIONS FRAMEWORK 9 2.1.2 QUALITY ASSURANCE AND LEARNING OUTCOMES 9 2.1.3 STAFF TRAINING IN LEARNING OUTCOMES 10 2.1.4 INVOLVEMENT OF STUDENTS 10 2.2 IRELAND 11 2.2.1 NATIONAL STRATEGY FOR HIGHER EDUCATION AND LEARNING OUTCOMES 11 2.2.2 QUALITY ASSURANCE AND LEARNING OUTCOMES 12 2.2.3 STAFF TRAINING IN LEARNING OUTCOMES 12 2.2.4 CENTRE OF EXCELLENCE IN TEACHING, LEARNING AND ASSESSMENT 14 2.2.5 PLANNING FOR THE FUTURE 15 2.3 LITHUANIA 15 2.3.1 STAFF TRAINING 16 2.3.2 QUALITY ASSURANCE AND LEARNING OUTCOMES 17 3 CONCLUSIONS 18 ANNEX 1 – EXAMPLES OF MODULE LEARNING OUTCOMES 19 ANNEX 2 – EXAMPLES OF PROGRAMME LEARNING OUTCOMES 20 ANNEX 3 – FORMAT OF A ONE-DAY TRAINING SEMINAR ON LEARNING OUTCOMES FOR ACADEMIC STAFF IN LITHUANIA 21 REFERENCES 23 2 LIST OF BOXES BOX 1. CONTINUING PROFESSIONAL DEVELOPMENT EVENTS AT UNIVERSITY COLLEGE CORK, IRELAND ......................... 13 BOX 2. EVENTS ON IMPLEMENTING LEARNING OUTCOMES IN LITHUANIA .............................................................. 16 3 List of Abbreviations CIRTL Centre for the Integration of Research, Teaching and Learning ECTS European Credit Transfer and Accumulation System EFO Externally Funded Output EHEA European Higher Education Area NAIRTL National Academy for the Integration of Research, Teaching and Learning NQF National Qualifications Framework NUS Scotland National Union of Students of Scotland QAA Quality Assurance Agency for Higher Education (United Kingdom) SCQF Scottish Credit and Qualifications Framework SKVC Centre for Quality Assessment in Higher Education UCC University College Cork UK United Kingdom UKHEEU UK Higher Education Europe Unit 4 Executive Summary This report is an outcome of a World Bank Externally Funded Output (EFO) project financed by the British Embassy Minsk supporting evidence-based tertiary education1 policies for better employment in Belarus. It forms part of one of two EFO project components, which addresses the implementation of learning outcomes in higher education. It presents international experiences with introducing learning outcomes in the higher education sector, common challenges, and promising approaches to overcome these. International trends in education show a shift from the traditional teacher-centred approach to a student-centred approach, that is, the focus is not only on teaching but also on what the students are expected to be able to do at the end of the module (short course) or programme. Learning outcomes are defined as statements of what a student is expected to know, understand and be able to demonstrate after completion of a process of learning. Learning outcomes constitute one of the basic building blocks of the Bologna Process. Three case studies are considered in this report in order to illustrate strategies that have been successfully used to implement a learning outcomes framework for teaching, learning and assessment in these countries. The United Kingdom (UK) has been at the heart of the Bologna Process from the outset. Overwhelming support for the Bologna Process has been shown by the UK, and the UK's engagement with the rest of Europe has been strengthened through the Bologna Process. Scotland has been chosen as a representative UK case study due to its excellent national qualifications framework (NQF), quality assurance systems linked to learning outcomes and great commitment shown to staff training in the area of learning outcomes. Scotland has hosted some landmark conferences on the role of learning outcomes in the Bologna Process. Ireland has developed a National Strategy for Higher Education in which learning outcomes play a key role. It has closely aligned quality assurance procedures with learning outcomes and had played a leading role in staff training in learning outcomes. It has produced a book on learning outcomes that is widely used at international level and has set up a Centre of Excellence in teaching, learning and assessment in University College Cork (UCC). This university has a number of accredited programmes to enable staff to gain a qualification in Teaching and Learning in Higher Education. Lithuania is an excellent example of a country that is making real and tangible efforts to provide training of the highest quality in the area of learning outcomes to the academic staff in its higher education institutions. It has organised many conferences, seminars and training workshops and is one of the few countries that has also organised training for quality assurance administrative staff in the area of learning outcomes. Lithuania has an excellent quality assurance system for higher education, including a process of external assessment of all study programmes by international experts. The analysis of learning outcomes plays a key role in the overall evaluation of programmes. Some common characteristics of good practice among the three case studies are a recognition of the importance of learning outcomes in teaching, learning and assessment; a clear strategy to address the challenges being faced when introducing a learning outcomes framework into higher education institutions; and a commitment at national and institutional level to provide training of the highest possible standard to staff in the area of learning outcomes. 1 Throughout this note, the terms “tertiary education” and “higher education” are used interchangeably. 5 1 Introduction This report is an outcome of a World Bank Externally Funded Output (EFO) project financed by the British Embassy Minsk supporting evidence-based tertiary education policies for better employment in Belarus. 2 The work under the EFO project addresses a) approaches towards tracking graduates as a tool to understand better the relevance of higher education for graduates and their performance in the labor market, and b) the implementation of learning outcomes in higher education as a means for improving the quality of provision and ensuring clarity in the description of courses. This report forms part of the second component and presents international experiences with introducing learning outcomes in the higher education sector, common challenges, and promising approaches to overcome these. The findings of this report will feed into a second report under the EFO project that provides practical guidance for promoting the use of learning outcomes in Belarus. The overall aim of the Bologna Declaration (1999) is to improve the efficiency and effectiveness of higher education in Europe. One of the main features of this process is the need to improve the traditional ways of describing qualifications and qualification structures. As a step towards achieving greater clarity in the description of qualifications, all modules and programmes in higher education institutions throughout the European Higher Education Area (EHEA) are required to be written in terms of learning outcomes. International trends in education show a shift from the traditional teacher-centred approach to a student-centred approach, that is, the focus is not only on teaching but also on what the students are expected to be able to do at the end of the module (short course) or programme. Statements called learning outcomes are used to express what the students are expected to achieve and how they are expected to demonstrate that achievement. Learning outcomes are defined as statements of what a learner is expected to know, understand and be able to demonstrate after completion of a process of learning. Thus, learning outcomes focus on the achievements of the student rather than the objectives of the teacher when teaching a module or an entire programme. Since learning outcomes focus on what a student should be capable of doing at the end of a process of learning, learning outcomes must always be written with an active verb. Learning outcomes are of two types: module learning outcomes and programme learning outcomes. Some examples of module learning outcomes are shown in Annex 1, and some examples of programme learning outcomes are shown in Annex 2. From studying the examples of learning outcomes, the following three points emerge: 1. Learning outcomes may be written in the form of short, simple and uncomplicated sentences. 2. All learning outcomes begin with an active verb. 3. Learning outcomes can be written for all subject areas. It is important to distinguish between aims, objectives and learning outcomes. Aims and objectives are part of the teacher-centred approach to teaching and learning. Learning outcomes are part of the student-centred approach to learning. The aim of a programme is a broad general statement of teaching intention, that is, it indicates what the teacher intends to cover in a programme. 2 Declan Kennedy, Senior Lecturer in Science Education at University College Cork (UCC), Ireland, drafted this report with support from Vitus Puttmann, World Bank Consultant, and under the guidance of Nina Arnhold, Senior Education Specialist, and Denis Nikolaev, Education Specialist, both the World Bank. 6 Example of an aim: To give students a critical understanding of current debates and issues relating to science education. It may take the teacher a number of years to achieve this aim, that is, aims involve long-term planning by the teacher. The objective of a module or lecture is a specific statement of teaching intention, that is, it indicates one of the specific areas that the teacher intends to cover. Example of an objective: To give students an appreciation of the unique nature of carbon and its ability to bond to other carbon atoms. This objective could take less than an hour to achieve, that is, the objective involves short-term planning by the teacher. When writing aims and objectives, words such as know, understand and be familiar with are commonly used. In contrast to aims and objectives, learning outcomes focus on the student and particularly on what the student can do at the end of the teaching process. Depending on the learning outcomes themselves, it may take a short time or a long time to achieve the learning outcomes. As mentioned above, when writing learning outcomes, active verbs must always be used. Terms such as know, understand and be familiar with should never be used when writing learning outcomes as these terms are part of the language of aims and objectives. When writing learning outcomes, it is important to bear a number of points in mind: • Learning outcomes must not simply be a “wish list” of what a student is capable of doing on completion of the learning activity. Learning outcomes must be based on the reality of what is happening during the teaching and learning activities. • Learning outcomes must be simply and clearly described so that all stakeholders (teachers, students, colleagues, external examiners, employers, quality assurance personnel, etc.) can understand them. • Learning outcomes must be capable of being validly assessed. Thus, every time a teacher writes a learning outcome, the teacher must ask one important question: “How will I assess this learning outcome?” Learning outcomes constitute one of the basic building blocks of the Bologna Process (Adam, 2008). Without learning outcomes, there can be no transparency or clarity in describing different programmes in various countries throughout the EHEA. Among the advantages of using learning outcomes are: • They assist curriculum design and help to clarify programme aims and module objectives. • They help to highlight the relationship between teaching, learning and assessment. • Students benefit from clear and precise statements of what they will be able to achieve after the specified period of study. • Students are provided with clear information about programmes and modules. • In terms of quality assurance, learning outcomes bring clarity and explicit transparency between qualifications and within individual qualifications. • Learning outcomes facilitate mobility of students and graduates seeking employment. 7 • Learning outcomes promote the continuum of education by providing a seamless link between the various levels of national qualifications frameworks (NQF). • Learning outcomes facilitate credit transfer and recognition of qualifications as a common language is used for describing programmes. In the same way that European Credit Transfer and Accumulation System (ECTS) credits have become the common currency for education in the EHEA, learning outcomes have become the international language for education. Assessing the extent to which learning outcomes have been achieved by students is far more challenging than simply measuring inputs or processes. Measuring learning outcomes needs a fundamental understanding of teaching, learning and assessment. It also requires strategies to ensure that learning outcomes, teaching and learning activities and assessment are aligned with each other (Biggs, 2003; Kennedy, 2007). Information on the precise state of implementation of a learning outcomes framework across the 48 countries that are part of the Bologna Process is difficult to assess. This is due to the fact that many national reports concentrate on the situation at national level without giving details of the situation at institutional level. One of the greatest challenges to ensuring the successful implementation of the Bologna Process is at the level of the higher education institutions themselves. There needs to be a transformation from the traditional input model for teaching and learning to setting up a dynamic equilibrium between teaching for understanding and achievement of the learning outcomes by students. Therefore, introducing a learning outcomes framework at the level of higher education institutions requires a carefully planned strategy driven by quality assurance and quality enhancement. It is hoped that the models of good practice highlighted in this report will help the higher education community of Belarus to achieve outstanding success in its efforts to successfully implement a learning outcomes framework in teaching and learning throughout the country. 2 Selected Case Studies This chapter takes a close look at three countries (UK – Scotland, Ireland and Lithuania) which are recognised internationally as being successful in implementing the requirements of the Bologna Process in terms of learning outcomes. A brief overview of each country with respect to key areas of success in the field of learning outcomes is given and discussed. 2.1 UK – Scotland The UK has been at the heart of the Bologna Process from the outset (Reilly, 2017). In 1996, the UK, France, Germany and Italy agreed the Sorbonne Declaration, which established the framework for the Bologna Declaration signed in 1999 by 27 countries. A study of the literature (UKHEEU, 2010) clearly demonstrates that overwhelming support for the Bologna Process has been shown by UK ministers for higher education, the Higher Education Funding Councils, the Quality Assurance Agency for Higher Education, the National Union of Students and the higher education institutions themselves in all parts of the UK. It is very clear that the UK's engagement with the rest of Europe has been strengthened through the Bologna Process. 8 The United Kingdom (UK) is a full member of the Bologna Process and EHEA since 1999 (EHEA, 2018). Scotland is part of the UK and has a population of 5.2 million. Universities Scotland is the representative body of Scotland's 19 higher education institutions and has responsibility for developing policy on behalf of the university sector. 2.1.1 National Qualifications Framework The Scottish Credit and Qualifications Framework (SCQF) is the NQF for Scotland. This framework is a learning outcomes-based structure with defined levels of qualifications, qualification descriptors and credits (National Bologna Report Scotland, 2013). The Scottish Credit and Qualifications Framework Partnership has a number of resources available to provide assistance in the area of learning outcomes related to qualifications and recognition of prior learning. Stephen Adam, a UK expert on the Bologna Process, summarised the situation in Scotland very well when he stated that “the most highly developed systems - that use learning outcomes as a basis of their qualifications frameworks, level descriptors, generic qualification descriptors, subject descriptors and at the level of individual modules - exist in Scotland and Eire (Republic of Ireland). It is no coincidence that these are the first two countries to have successfully undertaken the Bologna self-certification process where their national qualifications frameworks were articulated against the overarching framework of the qualifications of the EHEA” (Adam, 2008). The SCQF established in 2006 is a well-designed framework and may be considered as an example of international best practice in the design of qualifications frameworks in terms of structure and clarity. The framework satisfies all the standards of excellence expected of frameworks as it: • Assists students to easily plan their routes through the educational system. • Helps employers to understand the different types of qualifications. • Enables education and training providers to identify the level that a student has studied in a particular subject. • Makes it easier to transfer credits between different educational programmes. • Provides recognition of a wide range of programmes. 2.1.2 Quality Assurance and Learning Outcomes Quality Assurance Agency Scotland is part of the UK-wide Quality Assurance Agency for Higher Education (QAA). This organisation develops and publishes the NQF and some 50 or so subject benchmark statements that set out the characteristics and learning outcomes associated with programmes in each of the subject areas. QAA reviews are conducted by a team of QAA-trained academic peers and usually last a week. The review team considers how well the institutions are managing the programmes in terms of quality and standard of awards with reference to the NQF. In addition, all higher education institutions in the UK operate with a system of external examiners at the programme level. 9 2.1.3 Staff Training in Learning Outcomes The UK has played a pivotal role in educating academics, administrators and students about the key aspects of the Bologna Process – and particularly on the role played by learning outcomes in that process. As is the case with many countries, policy on staff training at institutional level is decided at institutional level. However, this training is usually compulsory for new probationary lecturers (National Bologna Report Scotland, 2013). One of the reasons for the success of the implementation of a learning outcomes framework in Scotland may be clearly identified from the fact that Scotland was very proactive from an early stage in organising conferences in the area of learning outcomes. Two important conferences were held in Scotland on the role of learning outcomes in the Bologna Process. These took place in 2004 and 2008 in Edinburgh. The documentation that arose from these conferences served as catalysts to guide institutions throughout the UK in implementing a learning outcomes framework in teaching, learning and assessment within their institutions (Edinburgh, 2004; 2008). One of the most important recommendations to arise from one of the above mentioned conferences was in the area of the implementation of learning outcomes: The seminar drew attention to the danger of learning outcomes being implemented in a false or superficial way in response to external pressures and the need to recognise that such a complex and multidimensional reform cannot be easily or rapidly achieved. If it is to be more than a merely cosmetic or bureaucratic exercise and if it is to result in a better learning experience for Europe’s students, the learning outcomes based approach needs time to develop, embed itself and mature in a way that respects and reflects the local priorities, diverse needs and national traditions of the Bologna countries. Training in the writing and implementation of learning outcomes should be given a high priority. (Edinburgh, 2008) In short, the key to the successful implementation of a learning outcomes framework lies in the training of staff. 2.1.4 Involvement of Students Scotland recognised the need to find effective ways of engaging stakeholders in the development and implementation of a learning outcomes framework for teaching, learning and assessment. Scotland put student-centred learning at the heart of the introduction of learning outcomes and involved students from the earliest stages onwards. The National Union of Students (NUS) of Scotland was closely involved in the Edinburgh conference where one of the keynote speakers was Jill Little, Deputy President of NUS Scotland. Her address (Little, 2008) clearly outlined the advantages of learning outcomes from the perspective of students: • Learning outcomes allow a more accurate picture of the learning experience. • Learning outcomes allow more transparency of degrees and modules. This greater transparency allows students to have greater ownership of their own educational journey. • Learning outcomes make it easier to obtain recognition of modules and degrees, and hence allow greater mobility for students. 10 • Learning outcomes enhance employability as a result of the clearer information deriving from using the language of learning outcomes. This is particularly useful when seeking employment abroad. • Learning outcomes provide employers with a better appreciation of the student's learning experience. • Learning outcomes encourage student-centred learning by focussing on student needs and on what students should be able to do when they achieve the learning outcomes. These ideas are also echoed by Aaron Porter, Vice-President (Higher Education), NSU Scotland, who stated that “the focus on the learning outcomes within qualifications will ensure greater emphasis on the skills necessary for the workplace and further enhance student employability” (UKHEEU, 2010). 2.2 Ireland Ireland is a full member of the Bologna Process and EHEA since 1999. Ireland has a population of 4.8 million and there are 7 universities and 15 institutes of technology in the country. All of these higher education institutions award degrees up to doctoral level. Ireland is one of the most successful European countries in implementing the reforms brought about by the Bologna Process (National Qualifications Authority of Ireland, 2009). It was the first country in the EHEA to establish a NQF based on learning outcomes, which was introduced in 2003. A number of reasons for the success of the implementation of a Learning Outcomes Framework in Ireland may be clearly identified from the literature (for example, Department of Education and Skills, 2011; National Bologna Report Ireland, 2013; National Bologna Report Ireland, 2016). 2.2.1 National Strategy for Higher Education and Learning Outcomes Since the introduction of the Bologna Process to higher education in Ireland, there has been a co- ordinated and strong support structure in place to ensure the successful implementation of the Bologna Process reforms in terms of introducing learning outcomes to the higher education sector. This successful implementation is highlighted and summarised in the report “National Strategy for Higher Education to 2030” (Department of Education and Skills, 2011). Among the key points implemented by the Department of Education and Skills were: • In 2005, all higher education institutions were directed to implement the Bologna Process reforms in terms of learning outcomes. All programmes and modules had to be rewritten to include emphasis on learning outcomes. This also applied to doctoral programmes, which were supposed to include training in the more generic and transferable skills to adapt to the demands of the workplace in the future. • Programmes in all higher education institutions in Ireland have been successfully modularised and semesterised and are described in terms of learning outcomes. • It was specified that programmes in all higher education institutions in Ireland must be designed to ensure that there is alignment between learning outcomes, teaching and learning activities, and assessment. • In planning for the future, key skills for graduates need to be expressed in terms of learning outcomes – particularly at programme level (programme learning outcomes). 11 2.2.2 Quality Assurance and Learning Outcomes In 2006, Ireland became the first European country to verify the compatibility of its NQF (based on learning outcomes) with the Bologna Process Framework in the EHEA (Irish Universities Association, 2007). The NQF involved the establishment of agreed learning outcomes, clearer routes of transfer and progression to enable a continuum of education for all students. This had enabled the inclusion of a number of innovations in higher education programmes. The implementation of a learning outcomes framework has greatly facilitated the creation of customised programmes to include modules from different institutions and across different disciplines. One of the reasons put forward for Ireland's success is the high quality assurance process that exists in higher education in Ireland. A report for the European Commission (2013) includes Ireland among countries that score highly on indicators that measure progress in quality assurance in higher education. The inclusion of learning outcomes in the description of all modules and programmes enables students to understand what they will be required to achieve in order to graduate. QQI (Quality and Qualifications Ireland) is an independent state agency responsible for promoting quality and accountability in education and training services in Ireland. It was established in 2012 and has responsibility for maintaining and developing the Irish NQF. It also has responsibility for approving programmes in higher education institutions and provides very useful information to assist academic staff tasked with the job of mapping programme learning outcomes on to the appropriate level of the NQF. The importance of linking quality assurance and learning outcomes cannot be over-emphasised. Improved levels of quality assurance mean more transparent documentation on modules and programmes, clarity of learning outcomes to be achieved, higher quality teaching, and more appropriate assessment techniques linked to learning outcomes. All programmes in higher education have external examiners appointed to monitor standards and make suggestions as to how the learning outcomes map on to the NQF. However, Ireland is not complacent in the area of quality assurance and in the National Strategy for Higher Education to 2030 (Department of Education and Skills, 2011), it is recommended that an independent review should be undertaken of the external examiner system and the overall grading system. 2.2.3 Staff Training in Learning Outcomes One of the key strategies implemented in Ireland to assist in the successful implementation of a learning outcomes framework in higher education was the recognition of one fundamental fact: academic staff in universities and institutes of technology are subject specialists and the majority of them do not hold a qualification in teaching, that is, they had never studied Education as a subject or undertaken any formal course in teacher training. Hence, the concept of learning outcomes in education, the role of learning outcomes in education, the linking of learning outcomes to teaching and learning activities and to assessment, the mapping of programme learning outcomes to NQFs, and the place of learning outcomes in the Bologna Process were all new areas to the vast majority of those teaching in higher education institutions. 12 It was also recognised in Ireland that, if it were left entirely to each institution to organise the training of its own academic staff in the area of learning outcomes, the quality of training could lack uniformity and be of varying standards. Box 1. Continuing Professional Development Events at University College Cork, Ireland In the aftermath of an international conference on implementing learning outcomes in higher education at University College Cork (UCC), Ireland, in 2006, several forms of support for academics in this field were initiated: • Staff training seminars in learning outcomes were held on a regular basis. These were organised by the Quality Promotion Unit of the university, were held at lunch time and were presented by Declan Kennedy. Free refreshments were provided – thus, the academic staff learned as they drank coffee and enjoyed a sandwich. • Arising out of the seminars, it was clear there was a need to produce a simple guide to learning outcomes which would be suitable for those who would not have a formal qualification in the field of education. The Quality Promotion Unit obtained funding from a government agency called the Higher Education Authority to publish the book Writing and Using Learning Outcomes: A Practical Guide (Kennedy, 2007). A hard copy was sent to every academic in the university and thousands of copies were distributed worldwide in thirteen languages. This publication is also available online for free.3 • UCC became a Centre for Excellence in the area of learning outcomes in Ireland and presented numerous workshops to staff in other higher education institutions in the country (for details about this development see Section 2.2. 4). Source: Authors. One of the first international conferences on implementing learning outcomes in higher education was held in University College Cork (UCC), Ireland, in February 2006. The theme of the conference was Implementing Learning Outcomes: Implications for Re-Defining Teaching and Learning. This conference was attended by large numbers of delegates from Ireland and many other countries from around the world. It had the effect of alerting staff in the higher education sector in Ireland of the movement towards the adoption of learning outcomes at international level. The conference served as a catalyst to initiate a series of continuing professional development events for academics (see Box 1). A number of important lessons were learned from these training sessions: • The introduction of a learning outcomes framework into higher education institutions is a process rather than an event. It must be carried out in stages and in an iterative fashion, that is, over a number of years. Teachers in universities need to assess the effectiveness of the learning outcomes written in the light of implementing and assessing these learning outcomes. • In Ireland a “bottom-up” approach was adopted for all programmes currently being taught, that is, learning outcomes for existing modules were first developed; then these module learning outcomes were mapped onto programme learning outcomes; the programme learning outcomes were then mapped onto the NQF of Ireland. 3 https://cora.ucc.ie/bitstream/handle/10468/1613/A%20Learning%20Outcomes%20Book%20D%20Kennedy.pdf?sequ ence=1 13 • For new programmes, a “top-down” model is used, that is, programme learning outcomes are developed from the NQF, then modules are designed around the programme learning outcomes, and finally module learning outcomes are drafted for every module. • The training seminars on learning outcomes served as a catalyst for encouraging staff who were interested in learning more about teaching, learning and assessment to enrol in a formal continuing professional development program in order to gain a qualification in Teaching and Learning. 2.2.4 Centre of Excellence in Teaching, Learning and Assessment In 2007, the Higher Education Authority of Ireland, funded by the Irish government, established the National Academy for the Integration of Research, Teaching and Learning (NAIRTL) at University College Cork (UCC). In addition to UCC being the lead partner, the NAIRTL partnership encompassed Cork Institute of Technology, Trinity College Dublin, National University of Ireland Galway, and Waterford Institute of Technology, all in Ireland. This management board co-ordinated teaching and learning initiatives in 38 higher education institutions. In Ireland, each higher education institution has its own teaching and learning centre. Funding was supplied under a special fund called the Strategic Innovation Fund, whose broad aim was to enhance quality assurance and improve the student learning experience at undergraduate and post- graduate levels in higher education institutions throughout Ireland. NAIRTL played a key role in supporting the various workshops and seminars on learning outcomes and provided research grants to enable teaching staff to research their own teaching practice and disseminate national and international examples and models of good practice. In UCC, the work of NAIRTL has been continued and expanded upon by the Centre for the Integration of Research, Teaching and Learning (CIRTL).4 This had broadened the scope of learning outcomes to assist academic staff in a number of ways: • Foster and conduct research into teaching and learning in higher education; • Encourage dialogue and make teaching and learning a scholarly activity; • Support and develop teaching, learning and assessment across all disciplines; • Encourage the development of innovative approaches to teaching, learning and assessment; • Identify and develop appropriate structures within universities to support teaching and learning. One of the most important roles played by CIRTL is the development of accredited qualifications for staff, who may undertake online courses – all of which involve lectures in the area of learning outcomes. The three programmes are: • Online Postgraduate Certificate in Teaching and Learning in Higher Education; • Online Postgraduate Diploma in Teaching and Learning in Higher Education; • Master’s Degree in teaching and Learning in Higher Education. Full details of these programmes may be found online (CIRTL Programmes, 2018). To illustrate the type of research undertaken for Master’s degree programmes, one of the professors in the faculty of engineering carried out research on the introduction of learning outcomes into the faculty. Among the research findings was that students liked the fact that he clearly explained the learning 4 https://www.ucc.ie/en/cirtl/aboutcirtl/ 14 outcomes of the programme to the students at the earliest possible opportunity (Fitzpatrick, Byrnes and Kennedy, 2009). In summary, the original “one-off” seminars and workshops on learning outcomes have now been developed into accredited programmes to give staff a qualification in Teaching and Learning in Higher Education. To date, 70 percent of the teaching staff at UCC hold a qualification in Teaching and Learning in Higher Education. This figure now sets the benchmark for other universities across Ireland. 2.2.5 Planning for the Future In Ireland, the government has developed a national strategy for higher education until 2030 (Department of Education and Skills, 2011). The National Forum for the Enhancement of Teaching and Learning in Higher Education became fully operational in October 2013. The role of the Forum is to enhance the teaching and learning for all students in higher education. Engaging with leaders, managers, teachers and students, the forum mobilises expertise and inputs from across the entire sector to extend and shape best practice in all institutes of higher education in Ireland. The forum is the key infrastructure for the enhancement of teaching and learning in Irish higher education, and for the implementation of the relevant recommendations of the National Strategy for Higher Education to 2030. Under the guidance of its board, whose members are chosen to represent the different areas of expertise in teaching and learning, the forum engages in a range of activities aimed at: • Championing all those who contribute to great teaching and learning in higher education; • Inspiring great practice, by celebrating examples of teaching that have a strong and positive impact on learning; • Identifying and promoting best practice in professional development. Recommendations include providing all students with the highest quality teaching and learning experience; integrating research with teaching and learning; and strengthening the focus on learning outcomes to ensure that all graduates acquire the key transferable skills that are needed in the economy and society of the twenty-first century. 2.3 Lithuania Lithuania regained its independence in 1990 and is a full member of the Bologna Process and EHEA since 1999. The Lithuanian NQF was established in 2010. Lithuania has a population of 2.9 million and there are 23 universities in the country. The higher education system in Lithuania is an interesting example of a country where the Bologna Process reforms began to be implemented to a certain extent at the structural level before the country joined the European Union in 2004. Research carried out in some higher education institutions in Lithuania shows that the Bologna Process reforms are bringing about real change in these institutions (Leisyte et al., 2009). 15 2.3.1 Staff Training Lithuania is an excellent example of a country that is making real and tangible efforts to provide training of the highest quality in the area of learning outcomes to the academic staff in its higher education institutions. In planning this training, it has identified some of the key issues and challenges faced by countries in implementing a Learning Outcomes Framework for Teaching and Learning. It is furthermore one of the few countries that has also organised training for quality assurance administrative staff in the area of learning outcomes. Topics covered by the various conferences, seminars and training workshops organised in Lithuania comprise (for a list of selected events see Box 2): introducing learning outcomes at a higher education institution; the practice of writing and using learning outcomes; linking learning outcomes to teaching and learning activities and to assessment; and covering the role of learning outcomes in quality assurance processes. Box 2. Events on Implementing Learning Outcomes in Lithuania • Preparing for the Bologna Process and the European Higher Education Area at Vytautas Magnus University (2010) • Introducing Learning Outcomes into Vytautas Magnus University – Challenges and Opportunities (2011) • Writing and Using Learning Outcomes (2012) • Training Seminar on Learning Outcomes for Co-Ordinators of the Evaluation Process of Higher Education Programmes in Lithuania (2013) • Undertaking Quality Review of Learning Outcomes in Higher Education Programmes in Lithuania (2013) • Implementing Learning Outcomes in Higher Education. Linking Learning Outcomes to Teaching and Learning Activities and to Assessment (2014) • The Role of Learning Outcomes in the Development of Student-Centred Teaching, Learning and Assessment within the Bologna Learning Network (2017) • Assessment of Learning Outcomes – Opportunities and Challenges in Higher Education (2017) Source: Authors. Lithuania furthermore was one of the first countries to pioneer the concept of the “one -day training seminar” for its academic staff. The format of this one-day training seminar was synthesised out of various training seminars presented by Declan Kennedy. The main topics addressed during this training include (for the agenda of the training see Annex 3): 1. The fundamentals of learning outcomes (for example, the meaning of the concept, its difference from other concepts, and the role of learning outcomes within the Bologna Process); 2. The practice of writing module and programme learning outcomes (including work based on the actual programmes of participants); 3. The relation between learning outcomes and competences; 4. The link between learning outcomes, teaching and assessment. 16 One of the ongoing challenges experienced in Lithuania (and many other countries) is the training of academic staff in moving from the old competence-based frameworks to learning outcomes-based frameworks for teaching, learning and assessment. 2.3.2 Quality Assurance and Learning Outcomes Higher education reform began in Lithuania in 1990. Lithuania has a particularly well-developed quality assurance system for higher education. The Centre for Quality Assessment in Higher Education (Studijų Kokybės Vertinimo Centras, SKVC) was founded in 1995 (SKVC, 2018) and has three main areas of responsibility: • To organise the evaluation of existing study programmes and proposed new programmes in higher education institutions; • To organise the evaluation of higher education institutions; • To provide a consultative service to higher education institutions regarding the recognition of qualifications obtained outside Lithuania. The process of external assessment of all study programmes by international experts was initiated in 1997. The analysis of learning outcomes plays a key role in the overall evaluation of programmes. In particular, efforts are made to establish if the quality of the learning outcomes corresponds to the general requirements of higher education programmes internationally: • By means of laws and regulations, national policy requires that procedures for student assessment must focus on learning outcomes (National Bologna Report Lithuania, 2016). • Quality assurance procedures for the external evaluation and accreditation of programmes require learning outcomes to be included in all these programmes (National Bologna Report Lithuania, 2016). • By law, SKVC procedures on external evaluation of all programmes as well as reviews by the higher education institutions themselves (internal quality assurance reviews) require that students be involved in the evaluation process (National Bologna Report Lithuania, 2016). Feedback is regularly obtained from students by means of surveys. Responsibility for obtaining this feedback rests with university senates or academic councils of higher education institutions. • Employers are regularly involved in curriculum development and drafting of learning outcomes. External evaluation reports of programmes by international experts have praised this aspect of quality assurance in higher education institutions in Lithuania. Lithuania is a good role model of a country that is making great efforts to address the challenges of the Bologna Process in the area of learning outcomes. Staff in higher education institutions have shown great enthusiasm in embracing the concept of learning outcomes. The progress being made is steady and substantial. Lithuania is putting a strong focus on moving away from the structures of the past, which concentrated on input factors such as duration and content of higher education qualifications, towards focussing on what students should know, understand and be able to do at the end of a learning process, that is, on student-centred education. Great credit is due to SKVC, which is one of the driving forces behind the progress being made. 17 3 Conclusions It is clear that there are many examples of good practice in the area of learning outcomes to be found in each of the three countries examined in this report. Among these common factors are: • A clearly described NQF to assist in the drafting of programme learning outcomes that map onto this framework. • A recognition of the importance of learning outcomes as the foundation stone on which the Bologna Process reforms are built. • A recognition of the link between learning outcomes and quality assurance systems at institutional level in terms of transparency and comparability of standards. • A recognition of the challenges to be faced and a clear strategy to address these challenges to successfully introducing learning outcomes frameworks into higher education institutions. • A recognition of the importance of involving all stakeholders (including employers and students) in the successful development and implementation of a learning outcomes framework. • A commitment at national and institutional level to providing training of the highest possible standard to staff in writing and using learning outcomes as part of an overall learning outcomes framework in teaching, learning and assessment. • Each country discussed in this report continues to build on good practices and to plan a strategy to further develop learning outcomes within higher education. The introduction of a learning outcomes framework into higher education is not a one-off event. It is an iterative process that takes several years and involves reflection, analysis, revision and a sincere commitment to best practice in teaching, learning and assessment. “The best learning outcomes are the product of sincere reflection about realistic and attainable combinations of knowledge and understanding, practical and cognitive skills, levels of autonomy, learning skills [.......] learning outcomes are challenging but it is impossible to have a meaningful European Higher Education area without their widespread and consistent use" (Adam, 2008). 18 Annex 1 – Examples of Module Learning Outcomes These examples of learning outcomes are drawn from a variety of different modules across different subject areas. It is common practice to insert the phrase «At the end of this module, students should be able to:» before listing the module learning outcomes for a particular module. • Define what behaviours constitute unprofessional practice in the solicitor – client relationship. • Outline the history of the Celtic peoples from the earliest evidence to the insular migrations. • Describe the processes used in engineering when preparing a design brief for a client. • Recall the axioms and laws of Boolean algebra. • Translate short passages of contemporary Italian. • Explain the social, economic and political effects of World War I on the post-war world. • Differentiate between civil and criminal law. • Apply knowledge of infection control in the maintenance of patient care facilities. • Show proficiency in the use of vocabulary and grammar, as well as the sounds of the language in different styles. • Compare and contrast the different electronic business models. • Recognize trends in atomic radii in the Periodic Table of the Elements. • Recognize and formulate problems that are amenable to energy management solutions. • Summarise the causes and effects of the 1917 Russian revolutions. • Propose solutions to complex energy management problems both verbally and in writing. • Evaluate marketing strategies for different electronic business models. • Appraise the role of sport and physical education in health promotion for young people. • Display a willingness to communicate well with patients. • Relate to participants in an ethical and humane manner. • Resolve conflicting issues between personal beliefs and ethical considerations. • Embrace a responsibility for the welfare of children taken into care. • Operate the range of instrumentation specified in the module safely and efficiently in the chemistry laboratory. • Perform a comprehensive history and physical examination of patients in the outpatient setting and the general medical wards, excluding critical care settings. 19 Annex 2 – Examples of Programme Learning Outcomes The programme learning outcomes listed below are from a four-year Bachelor of Science Education programme. On successful completion of this programme, students should be able to: • Recognise and apply the basic principles of classroom management and discipline. • Identify the key characteristics of excellent teaching in science. • Develop comprehensive portfolios of lesson plans that are relevant to the science curricula in schools. • Evaluate the various theories of teaching and learning and apply these theories to assist in the creation of effective and inspiring science lessons. • Critically evaluate the effectiveness of their teaching of science in the secondary school system. • Display a willingness to co-operate with members of the teaching staff in their assigned school. • Foster an interest in science and a sense of enthusiasm for science subjects in their pupils. • Synthesise the key components of laboratory organisation and management and perform laboratory work in a safe and efficient manner. • Communicate effectively with the school community and with society at large in the area of science education. 20 Annex 3 – Format of a One-Day Training Seminar on Learning Outcomes for Academic Staff in Lithuania 8.30 – 9.00 Registration Presentation 1 Learning Outcomes - the International Language for describing programmes in Higher Education Institutions 9:00 – 9:45 This presentation covers: • The meaning of learning outcomes. • The distinction between, aims, objectives and learning outcomes. • What is the Bologna Process all about? • What is a Learning Outcomes framework? • The international dimension of learning outcomes. • Resources for training academic staff on learning outcomes. • Case Study of staff training : University College Cork. • The quiet revolution of the Bologna Process. • Framework of qualifications for EHEA - Bologna Framework. • Distinction between generic descriptors and learning outcomes. • The Dublin Descriptors. • The challenge of ensuring that the National Framework of Qualifications is compatible with the Bologna Framework. 9:45 – 10:00 Questions and Answers 10:00 – 10:30 Coffee / tea Presentation 2 10:30 – 11:30 How do we write Module Learning Outcomes and Programme Learning Outcomes? This presentation covers: • Writing learning outcomes of international standard • The use of Bloom's Taxonomy in writing learning outcomes. • Examples of learning outcomes in the cognitive, affective and psychomotor domain. • What should a typical set of modules learning outcomes look like? • Linking learning outcomes and ECTS credits. • Challenges faced when writing learning outcomes. • The distinction between module learning outcomes and programme learning outcomes. • The two types of programme learning outcomes. • Examples of undergraduate and postgraduate learning outcomes. • Mapping programme learning outcomes on to the European Qualifications Framework. • Examples of generic phrases that can be used when writing programme learning outcomes. • Mapping programme learning outcomes on to module learning outcomes - some practical examples. • Importance of learning outcomes in programme accreditation. • Making programme learning outcomes explicit to students. • Benefits and potential problems of learning outcomes. 21 • Transnational implications of learning outcomes Workshop * Workshop on writing module learning outcomes 11:30 – 12:30 In this Workshop participants will get the opportunity to gain practice in writing module Learning Outcomes, discuss these learning outcomes with colleagues and investigate how these learning outcomes map on to Bloom's Taxonomy. 12:30 –1:30 Lunch Presentation 3 1:30 – 2:15 Learning Outcomes and Competences - How are they related? This presentation covers: • The meaning of the term competence (competency). • The many interpretations of the term competence. • Competence in terms of skill. • Competence in terms of knowledge and skill. • Competence as described in certain professions. • Confusion about generic competences. • Competence and competency. • Competence as an umbrella term. • The relationship between competences and learning outcomes. • Competence - a fuzzy concept. • Conclusions and recommendations. 2:15 – 2:30 Questions and Answers 2:30 – 3:30 Presentation 4 How do we link Learning Outcomes to Teaching and Learning Activities and to Assessment? This presentation covers: • Assessing learning outcomes using various forms of assessment. • The meaning of assessment. • Assessment and evaluation. • Common assessment techniques in higher education. • Multiple Intelligences Theory and Assessment. • The use of rubrics in assessing learning outcomes. • The work of Ramsden and Biggs in assessment. • Learning outcomes and constructive alignment. • Steps involved in linking Learning Outcomes, Teaching and Learning Activities and Assessment • Learning Outcomes and Level Descriptors on Qualification Frameworks • Details of module descriptions. • Getting everything right with learning outcomes - a process rather than an event. 3:30 – 4:00 Questions and Answers. Summary and Conclusions Presentation of Certificates 22 References Adam, S. 2008. “Learning outcomes current developments in Europe: Update on the issues and applications of learning outcomes associated with the Bologna Process.” In Proceedings of Bologna Seminar 2008. Edinburgh, Scotland. Biggs J. 2003. “Aligning Teaching and Assessing to Course Objectives.” Paper presented at the conference Teaching and Learning in Higher Education: New Trends and Innovations at the University of Aveiro, 13– 17 April 2003. CIRTL Programmes. 2018. https://www.ucc.ie/en/cirtl/staff/. Department of Education and Skills. 2011. National Strategy for Higher Education to 2030. Dublin Government Publications Sales Office. http://hea.ie/assets/uploads/2017/06/National-Strategy-for- Higher-Education-2030.pdf. Edinburgh. 2004. Proceedings of the UK Bologna Seminar - Using Learning Outcomes. http://www.aic.lv/bolona/Bologna/Bol_semin/Edinburgh/11_03_Edinb_Report.pdf. Edinburgh. 2008. Learning Outcomes Based Higher Education - the Scottish Experience. http://www.ehea.info/cid103095/learning-outcomes-edinburgh-february-2008.html. EHEA. 2018. “Members.” http://www.ehea.info/pid34250/members.html. European Commission. 2013. Quality Assurance in Higher Education: Ireland. https://eacea.ec.europa.eu/national-policies/eurydice/content/quality-assurance-higher-education- 31_sv. Fitzpatrick, J., E. Byrne, and D. Kennedy. 2009. “Making Learning Outcomes Explicit for Students of Process and Chemical Engineering.” Education for Chemical Engineers 4 (2009): 21–28. Irish Universities Association. 2007. A Framework for Quality in Irish Universities. Dublin: Irish Universities Association & Irish Universities Quality Board. Kennedy, D. 2007. Writing and Using Learning Outcomes – A Practical Guide. Quality Promotion Unit, University College Cork. https://cora.ucc.ie/bitstream/handle/10468/1613/A%20Learning%20Outcomes%20Book%20D%20Kenn edy.pdf?sequence=1. Leisyte, L., V. Navickiene, R. Zelvys, and L. Zenkiene. 2009. “The Understanding of the Bologna Process in Lithuanian Higher Education.” Paper presented at the 31st Annual EAIR Forum in Vilnius, Lithuania. Little, J. 2008. “Learning Outcomes: Why are they important for students?” In Proceedings of the Bologna Seminar: Learning Outcomes Based Higher Education - The Scottish Experience. http://www.ehea.info/cid103095/learning-outcomes-edinburgh-february-2008.html. National Bologna Report Ireland. 2013. National Report regarding the Bologna Process implementation 2009-2012 Ireland. http://www.ehea.info/media.ehea.info/file/Ireland/82/0/National_Report_Ireland_2012_567820.pdf. National Bologna Report Ireland. 2016. National Report regarding the Bologna Process implementation 2012-2015 Ireland. http://www.ehea.info/media.ehea.info/file/Ireland/82/2/National_Report_Ireland_2015_567822.pdf. 23 National Bologna Report Lithuania. 2016. National Report regarding the Bologna Process implementation 2012-2015 Lithuania. http://www.ehea.info/media.ehea.info/file/Lithuania/20/7/National_Report_Lithuania_2015_568207.p df. National Bologna Report Scotland. 2013. National Report regarding the Bologna Process implementation 2009-2012 UK Scotland. http://www.ehea.info/media.ehea.info/file/United_Kingdom/49/1/National_Report_UK- Scotland_2012_572491.pdf. National Qualifications Authority of Ireland. 2009. University awards and the National Framework of Qualifications (NFQ): Issues around the Design of Programmes and the Use and Assessment of Learning Outcomes. The University Sector Framework Implementation Network. Reilly, J. 2017. The UK in the European Higher Education Area. London: British Council. SKVC. 2018. “SKVC is Celebrating Its 20th Anniversary.” http://www.skvc.lt/default/en/news/skvc-is- celebratingits-20th-anniversary. UKHEEU. 2010. The European Higher Education Area: Celebrating a decade of UK engagement. London: UK Higher Education Europe Unit. 24