Document of The World Bank Report No: ICR2658 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-48090) ON A LOAN IN THE AMOUNT OF USD 40.4 MILLION TO ROMANIA FOR A KNOWLEDGE ECONOMY PROJECT August 21, 2013 Private and Financial Sector Development Unit Central/South Europe and Baltics (ECCU5) Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective June 24, 2013) Currency Unit = Leu 1.00 = US$ 0.289955 US$ 1.00 = 3.44882 RON FISCAL YEAR January 31-December 31 ABBREVIATIONS AND ACRONYMS CAS Country Assistance Strategy CIS Integrated System for the Issuance of Civil Information EBRD European Bank for Reconstruction and Development EC European Commission EU European Union FAs Family associations G2B Government to Business G2C Government to Citizens G2G Government to Government GOR Government of Romania ICT Information Communication and Technology KEP Knowledge Economy Project LCeNs Local Community e-Networks MAI Ministry of Administration and Interior MCRA Ministry of Culture and Religious Affairs MCIS Ministry of Communication and Information Society MERYS Ministry of Education, Research, Youth and Sport MPF Ministry of Public Finance MSMEs Micro and small and medium enterprises MTR Mid-term review NASME National Agency for Small and Medium Size Enterprises NTR National Trade Registry PAD Project Appraisal Document PAPIs Public Internet Access Points PDO Project Development Objective PMU Project Management Unit PPP Private-public-partnership R&D Research and development SEI Government’s National Computer-Based Education System program SEPs Self-employed persons SEN National Electronic System SMEs Small and medium enterprises STS Special Telecommunication Service TA Technical assistance Vice President: Philippe le Houerou Country Director: Mamta Murthi Sector Manager: Aurora Ferrari Project Team Leader: Arabela Aprahamian ICR Team Leader: Arabela Aprahamian ROMANIA Knowledge Economy Project CONTENTS 1. Project Context, Development Objectives and Design ................................................1   1.1 Context at Appraisal ............................................................................................. 1  1.2 Original Project Development Objectives (PDO) and Key Indicators ................. 2  1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification.............................................................................................. 3  1.4 Main Beneficiaries ................................................................................................ 3  1.5 Original Components ............................................................................................ 3  1.6 Revised Components ............................................................................................ 5  1.7 Other significant changes ...................................................................................... 7  2. Key Factors Affecting Implementation and Outcomes .............................................. 9  2.1 Project Preparation, Design and Quality at Entry ................................................. 9  2.2 Implementation ................................................................................................... 10  2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization .... 13  2.4 Safeguard and Fiduciary Compliance ................................................................. 15  2.5 Post-completion Operation/Next Phase .............................................................. 15  3. Assessment of Outcomes .......................................................................................... 16  3.1 Relevance of Objectives, Design and Implementation ....................................... 16  3.2 Achievement of Project Development Objectives .............................................. 17  3.3 Efficiency ............................................................................................................ 21  3.4 Justification of Overall Outcome Rating ............................................................ 22  3.5 Overarching Themes, Other Outcomes and Impacts .......................................... 22  3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops ... 24  4. Assessment of Risk to Development Outcome......................................................... 25  5. Assessment of Bank and Borrower Performance ..................................................... 25  5.1 Bank Performance ............................................................................................... 25  5.2 Borrower Performance ........................................................................................ 26  6. Lessons Learned ....................................................................................................... 27  7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 29  Annex 1. Project Costs and Financing .......................................................................... 30  Annex 2. Outputs by Component ................................................................................. 35  Annex 3. Economic and Financial Analysis ................................................................. 49  Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 50  Annex 5. Beneficiary Survey Results ........................................................................... 52  Annex 6. Stakeholder Workshop Report and Results................................................... 79  Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 80  Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 91  Annex 9. List of Supporting Documents ...................................................................... 92  MAP .............................................................................................................................. 93  A. Basic Information Knowledge Economy Country: Romania Project Name: Project Project ID: P088165 L/C/TF Number(s): IBRD-48090 ICR Date: 08/21/2013 ICR Type: Core ICR MIN. OF PUBLIC Lending Instrument: SIL Borrower: FINANCE Original Total USD 60.00M Disbursed Amount: USD 40.26M Commitment: Revised Amount: USD 40.26M Environmental Category: C Implementing Agencies: Ministry of Communication and Information Technology Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 03/09/2004 Effectiveness: 01/25/2006 01/25/2006 06/24/2010 Appraisal: 03/01/2005 Restructuring(s): 08/10/2011 02/14/2013 Approval: 11/29/2005 Mid-term Review: 12/15/2008 11/23/2008 Closing: 06/30/2011 02/28/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Satisfactory Moderately Satisfactory Performance: Performance: i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) General industry and trade sector 22 7 General public administration sector 32 92 Other social services 4 Primary education 21 1 Tertiary education 21 Theme Code (as % of total Bank financing) Administrative and civil service reform 29 14 Education for the knowledge economy 29 79 Infrastructure services for private sector development 14 3 Micro, Small and Medium Enterprise support 28 4 E. Bank Staff Positions At ICR At Approval Vice President: Philippe H. Le Houerou Shigeo Katsu Country Director: Mamta Murthi Anand K. Seth Sector Manager: Aurora Ferrari Gerardo M. Corrochano Project Team Leader: Arabela Sena Aprahamian Gregory T. Jedrzejczak ICR Team Leader: Arabela Sena Aprahamian ICR Primary Author: Greta Minxhozi F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The objective of the Project was to accelerate the participation of knowledge disadvantaged communities in the knowledge-based society and economy in Romania. ii Specifically, in targeted disadvantaged communities, Local Community e-Networks (LCeNs) would evolve into a daily tool for education, business and public communications with the Government, with increasing private management and financial self-sustainability. To this end, the Project would enable the Borrower to implement the National Strategy for the New Economy and Implementation of the Information Society, and the European Union programs related to the information society for all. The project aimed to achieve this objective in a three stage approach: first, to provide access and internet connectivity through the establishment of Local electronic Community Networks (LCENs) with access nodes in local schools, public administration, public libraries and public point of access points (PAPIs); second, to provide technical assistance and matching grants to increase the digital and other competencies of beneficiaries and support firms to adopt e-business solutions and thus increase their participation in the information society, and third, to improve the quality of use and benefits through the provision of value added services (including through e-government services to citizens and businesses). Two outcome indicators were set to measure the PDO achievement. Thus the project envisaged that at least 40 percent of population in targeted communities used the LCeNs as a tool for education, business or public administration and was satisfied with the results, and at least 20 percent of the LCeNs operate as sustainable PPP (public private partnerships) arrangements. The PAD envisaged the project to target at least 200 communities exhibiting the greatest knowledge disadvantage. Revised Project Development Objectives (as approved by original approving authority) The Project Development Objectives were not revised during project implementation. The original PDO indicators were complemented in the 2011 (adoptive) restructuring by adding a new PDO indicator to reflect the scaled up and changed technical solution of the Integrated System for Civil Information Status (CIS) under the subcomponent 2.B. This PDO indicator ("Stage of the available online sophistication for the civil status e- Government services, in line with the National strategy on Information Society and EU Digital Agenda") was chosen to better reflect the availability at the highest level of sophistication of the G2C service (full two-way interaction between citizen and public administration) among other e-government services available through LCeNs. This PDO indicator was dropped in the 2013 restructuring that cancelled the financing of the Integrated Network for CIS in view of availability of European Funds and project closure (see Sections 1.6, 1.7, 2.2 and 3). (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Percentage of relevant population in k-disadvantaged communities that have used Indicator 1 : the LCeNs as a tool for education, business or public administration and are satisfied with the results iii Value 40 quantitative or 0 40 43 Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 108% achievement. Comment 1 Riga Agenda for IS: EU27(2005): 45% (incl. % RO(2005): 17.5%. RO (2011): 38.6 %. Target: halving the disparities (at 2005 achievement) level) between RO and EU27 (average) Indicator 2 : Percentage of LCeNs that operate as sustainable PPP arrangements Value quantitative or 0 20 20 30 Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 12/07/2012 Comments 150% achievement. In addition 51% of communities have projects approved (incl. % pending signing. 80% of the mayors elected in 2012 state that will cover the achievement) Internet cost and other operational costs for LCeNs. (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Number of K-disadvantaged communities with LCeNs operating satisfactorily Indicator 1 : towards a sustainable future in line with their business plan Value (quantitative 0 200 at least 200 229 or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 115% achieved. In add., 91% of the mayors elected in 2012 appreciate that PAPI Comments will maintain its current activities (65%) or will further develop and diversify its (incl. % activities (26%). Note: RP2011+2013 show incorrectly 255 as a target (=all achievement) selected comm.). Percentage of communities that have prepared satisfactory 3-year LCeN Indicator 2 : sustainability plan (2011-2013) Value (quantitative 0 90 90 90 or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 100% achievement ( 229 communities (or 90%) prepared sustainability plans out (incl. % of the 255 where LCeNs were established) achievement) Indicator 3 : Level of ICT integration in schools Value (quantitative 35 n/a 100 93 or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 93% achievement. The lower than target performance is attributable to (incl. % unobservable impact of the TA in schools. The respective sub-indicator is iv achievement) expected to be fulfilled and can be reported only at the beginning of the 2013- 2014 school year. Indicator 4 : Percentage of targeted teachers that are using the new skills in classroom Value (quantitative 58 75 75 75 or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 100% achievement. In 2012, 61.5 % of the teachers used computers more than (incl. % once a week (34.5%) and very often (27%), while 20.5% a few times monthly. achievement) Percentage of people trained in the LCeNs that aquired basic digital literacy Indicator 5 : skills and are using the new skills in the work Value (quantitative 0 75 75 85 or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 113% achievement. ICT training events were organized during 2012 for people (incl. % from different categories of beneficiaries (public administration (105), farmers achievement) (156), entrepreneurs (104), children (222), other category (87). Online registration of businesses available as result of the knowledge achieved Indicator 6 : under the project Value (quantitative no n/a yes yes or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 100%. The utilization of the on-line registr. system developed was stopped in (incl. % 2011 due to elegal changes. A new module was built based on project-financed achievement) SW and is operational since 2012. Indicator 7 : Readiness of bidding documents for the implementation of CIS with EU funding Value (quantitative no N/A yes yes or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments (incl. % 100% achievement. achievement) Indicator 8 : Number of visits in the e-Portal Value (quantitative 0 n/a 500000.00 1104995.00 or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 220% achievement. Reflects the cumulative number of visits, of which different (incl. % visitors 1,104,995.Revised indicator as part of August 2011 project restructuring achievement) Indicator 9 : Number of grants awarded Value (quantitative 0 300 41 41 or Qualitative) v Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments 100% achievement. Cancelled the loan funding in 2011 as EU grant funds were (incl. % available in an amount of about EUR 151 million to finance the grants. achievement) Indicator 10 : Volume of Bank Support: Institutional Development - SME Value (quantitative 0 n/a 1392191.00 1392191.00 or Qualitative) Date achieved 12/31/2005 06/30/2011 02/28/2013 02/07/2013 Comments Introduced to comply with World Bank requirements to report on relevant core (incl. % indicators. Not used as a performance indicator for the project. achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 12/11/2006 Satisfactory Satisfactory 1.96 2 12/21/2007 Satisfactory Satisfactory 15.19 3 04/24/2008 Satisfactory Satisfactory 22.85 4 12/30/2008 Satisfactory Satisfactory 24.03 5 12/10/2009 Satisfactory Satisfactory 26.64 6 09/26/2010 Satisfactory Satisfactory 31.25 7 01/16/2011 Satisfactory Satisfactory 31.72 8 07/12/2011 Moderately Satisfactory Moderately Satisfactory 33.57 9 08/21/2011 Moderately Satisfactory Moderately Satisfactory 33.57 Moderately 10 04/22/2012 Unsatisfactory 36.75 Unsatisfactory 11 08/19/2012 Moderately Satisfactory Unsatisfactory 37.54 Moderately 12 02/27/2013 Satisfactory 38.99 Unsatisfactory H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Reallocated US$ 10.5 million of savings (or 15% of the loan) among categories of 06/24/2010 N S S 30.16 expenditures, while maintaining the original allocation of the loan proceeds by components. vi ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Cancelled the second call of funds under the grant facility subcomponent 3B; expanded the scope of subcomponent 2B (est. costs increased to US$ 15 08/10/2011 N MS MS 33.57 million); increased the cost of subcomponent 1A for original activities; revised the monitoring indicators accordingly. Cancellation of two remaining activities under subcomponents 1A and 2B in the amount of 02/14/2013 N MS U 37.54 $19.6 million and dropping the related outcome and component indicators. I. Disbursement Profile vii 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Romania started the negotiations process for European Union (EU) in February 2000 and since had committed to adopt the EU directives and strategies related to the development and use of Information Communication and Technology (ICT) as a drive for economic modernization. The Government of Romania (GOR) established a Task Force chaired by the Prime Minister to lead this agenda. A National Strategy for the New Economy and the Implementation of the Information Society (2002-2010), built on the European Lisbon Strategy1 was prepared and emphasized the need to: improve and expand the ICT enabled infrastructure (broadband), including through public interventions for addressing the digital divide2 challenges; increase the use of e-business solutions by firms and of e-government services to increase the competitiveness of the firms and of the Romanian business environment; the development of human resources and in particular of digital competencies to be able to compete in the European and global markets; and, to achieve network security. The project was expected to use as benchmarks the objectives, indicators and targets set by the European Digital Action Plans, 3 supporting the implementation of the Lisbon Strategy. The Government’s National Computer-Based Education System program (SEI) was introduced in support of ICT education in upper secondary schools in 2001-2003. Under SEI, high schools were provided with: (i) computers, standard software, and connectivity equipment, plus technical support: (ii) educational software, content and knowledge management, administrative support software platforms; (iii) educational multimedia content; and (iv) training of teachers and administrators. The Government’s objective was to extend the penetration of computers to basic and lower secondary schools, building on the lessons of the SEI project, given that at the time only 20% of these schools had the desired level of equipment and trained teachers. Romania was ranking behind other new EU members countries in terms of e-business adoption by firms and the availability and uptake of e-government services by citizens and businesses. The National Electronic System (SEN), launched in 2003, was designed to increase the efficiency in the interaction with public administration, reduce costs for both public and private entities and citizens, limit the potential for corruption, and increase public trust in the administration. Except for the e-government system for public procurement, other key 4 e- government services were not available nation-wide or were available at very basic sophistication 1 The Lisbon Strategy, also known as the Lisbon Agenda or Lisbon Process, was an action and development plan devised in 2000, for the economy of the European Union between 2000 and 2010 and called for transforming the EU into “the most competitive knowledge based economy by 2010, capable to sustain economic growth by creating new jobs and by the existence of an increased economic cohesion�. It was followed by the Europe 2020 Strategy adopted in 2010. 2 The digital divide refers to “the gap between people with effective access to information and communications technology (ICT), and those with very limited or no access�. The gap between people with Internet access and those without is one of many aspects of the digital divide. Whether someone has access to the Internet can depend greatly on financial status, digital competency, geographical location as well as government policies. 3 e Europe 2005 action plan (launched in 2002) as a pillar for Lisbon Agenda and aimed to stimulate the development of services, applications and contents while speeding up the deployment of secure broadband Internet access. There was also the general aim of providing access for everyone in order to combat social exclusion, whether it is due to particular needs or geographical disparities. It was continued with the i2010 initiative (launched in 2005) that set new targets for 2011 and by Digital Agenda for 2020 (launched in 2011) as pillar in the Europe 2010 strategy. 4 20 key e-government services G2C and G2B were defined as priority (12 for citizens and 8 for business). 1 level. Besides the development of the key e-government services major challenges were also to scaling-up pilots and to increase the businesses and citizen’s awareness of and uptake of such services. The Country Assistance Strategy (CAS) for Romania (2001) had identified accession to the EU as a key priority for development and the Knowledge Economy (KE) Project supported this objective by increasing the use of ICT technology as a means for economic development, in line with the European Digital Action Plans. Although there were few operations preceding the KE Project that related to knowledge economy activities5, none of them tackled information society challenges in a coherent, focused, and strategic manner. This was the underlying basis of the proposed Project and the Bank’s comparative advantage. Foreign donor activities were fragmented and pilot-based, and overall, lacking a systemic, cross-sector approach. It was assessed that the Bank, with its multi-sector approach, was able to advise the Government on balancing public goods with private initiative. 1.2 Original Project Development Objectives (PDO) and Key Indicators The objective of the Project was to accelerate the participation of knowledge disadvantaged communities in the knowledge-based society and economy in Romania. Specifically, in targeted disadvantaged communities, Local Community e-Networks (LCeNs) would evolve into a daily tool for education, business and public communications with the Government, with increasing private management and financial self-sustainability. To this end, the Project would enable the Borrower to implement the National Strategy for the New Economy and Implementation of the Information Society, and the European Union programs related to the information society for all. The project aimed to achieve this objective in a three stage approach: first, to provide access and internet connectivity through the establishment of Local electronic Community Networks (LCENs) with access nodes in local schools, public administration, public libraries and public point of access points (PAPIs); second, to provide technical assistance and matching grants to increase the digital and other competencies of beneficiaries and support firms to adopt e-business solutions and thus increase their participation in the information society, and third, to improve the quality of use and benefits through the provision of value added services (including through e-government services to citizens and businesses). Two outcome indicators were set to measure the PDO achievement. Thus the project envisaged that at least 40 percent of population in targeted communities used the LCeNs as a tool for education, business or public administration and was satisfied with the results, and at least 20 percent of the LCeNs operate as sustainable PPP (public private partnerships) arrangements. The PAD envisaged the project to target at least 200 communities exhibiting the greatest knowledge 6 disadvantage . 5 Private Institution Building Loan (P039251), the Public and Private Institution Building Loan (P069679), the Education Reform Project (P008784) and the Higher Education Reform Project (P008793). 6 Knowledge typologies for communities were defined as part of the project social assessment. PAD contains an additional reference (page 59) that about 45 percent of the communities exhibiting the greatest K-knowledge disadvantage would be eligible for inclusion in the project. 255 communities were thus selected for initial project interventions. 2 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification The Project Development Objectives were not revised during project implementation. The original PDO indicators were complemented in the 2011 (adoptive) restructuring by adding a new PDO indicator to reflect the scaled up and changed technical solution of the Integrated System for Civil Information Status (CIS) under the subcomponent 2.B. This PDO indicator ("Stage of the available online sophistication for the civil status e-Government services, in line with the National strategy on Information Society and EU Digital Agenda") was chosen to better reflect the availability at the highest level of sophistication of the G2C service (full two-way interaction between citizen and public administration)7 among other e-government services available through LCeNs. This PDO indicator was dropped in the 2013 restructuring that cancelled the financing of the Integrated Network for CIS in view of availability of European Funds and imminent project closure (see Sections 1.6, 1.7, 2.2 and 3). 1.4 Main Beneficiaries The project main beneficiaries as described in the Project Appraisal Document (PAD) were: (i) the population at large in the 255 targeted knowledge disadvantaged communities beneficiaries of the project interventions, representing about 1.09 million people; (ii) the schools and public libraries, the teachers and students within the selected communities; and (iii) the local public administrations of the selected communities and the LCeNs staff (managers, IT administrators); and (iv) the self-employed, family associations, micro, small and medium enterprises. In addition the staff of the Ministry of Communication and Information Society (MCIS), the Ministry of Public Administration and Interior (MAI), Ministry of Education, Research, Youth and Sport (MERYS), the National Agency for Small and Medium Size Enterprises (NASME), the Ministry of Culture and Religious Affairs (MCRA) and of the National Trade Registry (NTR) were indirect beneficiaries of the knowledge created under project, but also partners in the implementation of the project. 1.5 Original Components Component 1: Access to ICT in knowledge disadvantaged communities and improved digital literacy (US$ 43.1 million; loan proceeds US$ 37.0 million) The objective of this component was to provide to the people living in the targeted communities increased access to and meaningful use of ICT-enabled infrastructure and services through the supply of physical equipment and internet connectivity and provision of technical assistance. This expected to enhance the human and social capabilities of people in these communities and would have increased the benefits from using the LCeNs enabled services. This objective supported the National Strategy for Information Society’s objective to reduce the digital divide between targeted rural and small urban areas and the rest of the country and thus helped accelerate the participation of such knowledge disadvantaged communities in the knowledge-based society and economy in Romania. These objectives were to be achieved through three sub components: 7 No indicator to measure the uptake by citizens was proposed given the impact that could be achieved in the limited time until the end of the project. 3 Subcomponent 1.A: Improving access by establishing Local Community e-Networks (LCeNs). The purpose of this component was to help establish at least 200 multi-purpose local e-networks (LCeNs) that would serve as “knowledge-centers�, providing information, services and benefits to citizens and small businesses in selected knowledge disadvantaged communities. Subcomponent 1.B: Digital literacy for communities. This sub-component was to support activities which promote increased access to ICT and e-services via LCeNs. The focus was on how emerging ICT technologies can provide practical benefits, how to apply them to business processes, and how to find appropriate sources of help to do so. This would be helped through technical assistance and training programs. Subcomponent 1.C: Digital literacy for schools. This sub-component was to focus on the enhancement of digital competencies within primary and lower secondary education levels through the integration of the ICT into the classroom, thereby improving teachers and students’ skills and knowledge. This was to be achieved through the provision of technical assistance at national level for ICT in education policy formulation, and at local level, through the utilization of LCeN nodes that are based in schools and local public libraries. Component 2: Development and promotion of e-government services (US$ 8.8 million; loan proceeds US$ 7.0) The objective of this component was to enhance the accuracy and reduce the time of interaction between citizens and business and the Government through the deployment of two selected e-government services namely: (i) for the online business registration of family associations and self-employed and (ii) for civil status information and documentation. This objective was to be achieved through two sub-components: Sub-component 2A: Online system for notifications and authorizations of local businesses. This subcomponent was to help the development of an online system for the registration of family associations and self-employed individuals that, according to the legislation at the time, were to use local administrations as intermediaries in the registration process. The system would have enabled such businesses to get information on the relevant registration legislation and forms by accessing the sites of the public administration or of the National Trade Registry; in addition, local authorities would have used the internet to electronically send to and receive from the National Trade Registry the documentation and confirmation of business registration respectively, thus would have reduced the time of business registration. LCeNs nodes (Public Administration Points or PAPIs and local administration) were going to be used as access points. Sub-component 2B: Integrated network civil information and documents. This subcomponent would have allowed citizens to apply online and receive civil status certificates, such as birth and marriage certificates, and thus to ensure more efficient interactions with the public authorities and more accurate transactions and records of the civil status central data-basis of the Ministry of Administration and Interior. The system was to be initially implemented in the communities targeted by the project with the objective to scale it up to national level. Component 3: Promotion of e-commerce and innovation support for MSMEs (US$ 11.9 million; loan proceeds US$ 11.3 million) The objective of this component was to facilitate businesses access to knowledge through the development of an e-Portal for e-commerce and business networking and the adoption by 4 firms of innovative business solutions, including the ICT based. This objective would be achieved through two sub-components: Sub-Component 3.A: Portal for promotion of e-commerce and business networking. The e-Store portal was to provide MSMEs with relevant information, how-to tools, training, consulting, references and other resources for developing e-business competency, to share knowledge, and to establish contacts for commercialization of new technologies, the development of supply chains and business solutions. In particular, the e-Store was to assist in the development of opportunities in knowledge disadvantaged areas, by providing these communities and businesses with better access to information, foster socio economic development and give local producers access to market information. Subcomponent 3.B: Grant Facility. The Facility was to provide grants to individual small and medium enterprises, self-employed persons (SEPs) and family associations (FAs), enterprise associations and clusters (or consortia of these clients) to facilitate e-business adoption and improve the competitiveness of Romanian firms in domestic and foreign markets. Component 4: Project management This component was to finance the incremental operating costs of the Project Management Unit (PMU) within the Ministry of Communication and Information Society (MCIS), provision of vehicles and consultants services for audits and for the monitoring and evaluation of the project. 1.6 Revised Components Some components were revised through restructurings approved by Bank management, as described below, to adjust to changed circumstances at the request of the Borrower: Original component Revised component Comments Subcomponent 1A: Improving Access by Restructuring approved by Bank The changes did not concern the establishing Local Community e-Networks management in August 2011: project/component objectives, description or indicators. They concerned the Increased the cost of subcomponent 1A component costs and source of financing. for the original activity “TA for communities� based on The additional TA aimed to further the savings in Operating Costs Category support communities in the (USD 5.5 million) resulted from changed implementation of the LCeNs source of financing of LCeN staff salaries sustainability plans and to disseminate the and equipment replacement from the loan key lessons to other disadvantaged to local financing. communities in Romania. The savings arose because the 255 Local Authorities participating in the project committed to and financed the salaries of the LCeNs staff as part of the partnership agreement with MSIS to be beneficiary of the project. The savings also arose because the Government decided not to finance from loan proceeds the replacement of the LCeN equipment procured under the project, as envisaged in the PAD. The financing of the replacement was taken over by the local administrations. Restructuring approved by Bank The Government requested to cancel management in February 2013: US$4.5 million under subcomponent 1.A. because of the lack of Government budget Cancelled the TA for the above Extended allocation for the project and closure of TA for Communities activity in the the project. 5 Original component Revised component Comments procurement plan and the related allocation of US$ 4.5 million in the loan agreement. Subcomponent 2B: Integrated network Restructuring approved by Bank The changes did not affect the civil information and documents. management in August 2011: project/component objective/description Provision of equipment, consultants’ (besides for the change from using services and training to government Changed the approach for implementing internet to state intranet) since the original authorities, and provision of internet the CIS system and increased the project component aimed to finance full scale service for county authorities, for the cost allocation (the estimated cost of the implementation and deployment of the development of an Integrated Network for new approach were $15 million): (i) G2C for Civil Data being piloted through Citizen Status Information and nation-wide implementation in one stage the use of LCeNs as the primary means of Documentation (CIS ) as opposed to the gradual scaling up, as access, prior to full scale up deployment envisioned in the PAD (this did not at national level, in a phased approach; require any changes in the original and the provision of the G2C for CIS was description) and (ii) utilization of the state envisaged by the National Strategy for intranet managed by the Special Information Society. Communication Service (STS) as channel of communication between local administrations and the central government as opposed to internet, as provided by the PAD (this required removal of “provision of internet service for county authorities� from the description in the loan agreement). Added the STS in the Project Management (modified the PAD Project Management description) Added new PDO and Component intermediate indicators (1 to 6) to reflect the scaled up and changed technical solution of the G2C for CIS under the subcomponent 2.B ("stage of the available online sophistication for the civil status e- Government services, in line with the National strategy on Information society and EU Digital Agenda") and dropped original Component 2 intermediate indicator. Restructuring approved by Bank The Government requested to cancel management in February 2013: US$15.1 million because of the lack of Government budget allocation for the Changed description to “provision of project and closure of the project. After consultant services to government the cancellation, the subcomponent authorities for the development of an covered only the provision of consulting integrated network for citizen status services for designing the G2C for Civil information and documentation� and Data and did not finance the cancelled $15.1 million under the loan. implementation and deployment of the Dropped the PDO and Component 2 integrated nation-wide system though Intermediate indicators (1to 6) introduced provision of equipment and training to as part of the 2011 restructuring to reflect government authorities, and provision of the cancellation and added two new internet service for county authorities. component indicator to reflect the actual project investments. Subcomponent 3.B: Grant facility Restructuring approved by Bank The Government requested to finance Operation of a facility to provide co- management in June 2010: consultant services to increase the financing grants from the proceeds of the awareness and quality of project proposals Loan to eligible beneficiaries to facilitate Provision of consultant services was in the second call. This provision was to e-business adoption. introduced as eligible expenditure. mitigate the low capacity (knowledge and resources) of firms in the target communities to prepare and implement projects and make use of the project grant facility. This was achieved by reallocation among expenditure categories from Grant Facility category to consulting services 6 Original component Revised component Comments category within subcomponent 3.A Restructuring approved by Bank The Government requested the management in August 2011: cancellations of further calls under the Grant facility given the availability of EU Cancelled remaining activities under the grant funds in an amount of EUR 151 Grant facility, reallocated US$ 8.2 million million to finance the adoption of e- to Component 2B and reduced the target business solutions by firms. Savings were for the relevant Component 3 Intermediate reallocated to Component 2B to finance indicator from 300 to 41. the new design for CIS. Component 4: Project Management: Restructuring approved by Bank The change was necessary to allow the management in February 2013: financing of PMU expenditures related to project document archiving. Added “archiving of Project documents� under “Incremental Operating Costs� as an eligible expenditure. 1.7 Other significant changes Extension of the closing date: The closing date was extended by 20 months from June 30, 2011 to February 28, 2013 to allow the implementation of the original project activities while including additional sustainability measures aimed at mitigating the risks towards achieving the original PDO, as identified by project Mid-term review (MTR). The proposed changes fell under the original PDO. By MTR the first stage towards meeting the PDO (access) had been successfully achieved through the establishment of the LCeNs in 255 communities and the activities planned under the last two project stages (use – benefits) were in different stages of implementation. Four major original activities were still in the planning stage as they were following the general path and sequence corresponding to the staged approach envisaged under the project. The additional sustainability measures to support the original activities included: (i) intensified assistance to communities to reduce the large variation of ICT resource utilization and service delivery in LCeNs and to ensure their sustainability under Component 1A; (ii) intensified provision of technical assistance (TA) in schools for integrating the ICT in the education process, while addressing the sustainability of such interventions by anchoring them to the national policies and programs under subcomponent 1B; and (iii) provision of TA to businesses to increase the quality of project proposals in the second call under the grant facility subcomponent 3.B. The extension was also requested to implement the original CIS system under subcomponent 2.B due to significant delays (about 30 months) from the original timetable because of difficulties in agreeing on the technical and business specifications solution, mainly driven by security concerns expressed by the Ministry of Public Administration and Interior. The extension was approved because the subcomponent remained highly relevant for the country as its implementation contributed to achieving the commitments made under the European Agenda i2010 and correlated with other European projects such as the issuance and exchange of civil status documents and information among all EU member countries in view of free circulation of persons in the Shengen space. Reallocations: There were two reallocations of loan funds approved by Bank management in June 2010 and August 2011 (see Annex 1.c):  The restructuring approved in June 2010 reallocated US$ 10.5 million of savings (or 15% of the loan) among categories of expenditures, while maintaining the original allocation of the loan proceeds by components. The loan allocation for the operating costs was reduced from the original US$8.5 million to zero because the operational and maintenance costs of LCeNs under the component 1 were financed by local governments 7 or covered under existing contracts under the project (Section 1.6). Under the grant facility subcomponent 3.B, the loan allocation for grants was reduced from US$9 million by US$1.8 million because it became essential to mitigate with TA the low capacity (knowledge and resources) of firms to prepare and implement projects, further aggravated by the current economic context. The savings were used to finance additional sustainability measures described in the above paragraph.  The restructuring approved in August 2011 contained three major changes. First, the second call of funds under the grant facility subcomponent 3.B was cancelled because EU grant funds were available in an amount of about EUR 151 million to finance the adoption of e-business solutions by firms. The savings were reallocated to finance the expended scope of the subcomponent 2.B “Integrated network civil information and documents�. The proposed approach envisaged: (i) nation-wide implementation of the system in one stage (vs. gradual scaling up of the system, as envisioned in the PAD), implying that the project was to cover 3,210 Territorial Civil Offices compared to 255 KE Project communities planned for the initial stage; and (ii) use of state intranet vs. internet as a channel of communication between local administration and the central Government infrastructure (Government-to-Government (G2G) (Section 1.6). The total costs for implementing the new Integrated System for the Issuance of Civil Information (CIS) subcomponent country-wide were estimated to about US$ 15 million to cover the hardware and software equipment for 3,210 offices and for their connectivity to the intranet of the state (vs. the initial budget of US$ 8.8 million). The restructuring did not affect the objective of this subcomponent or the PDO since: (i) the project aimed to “finance the full scale implementation and deployment of the G2C service being piloted through the use of LCeNs as the primary means of access, prior to full-scale up deployment on a national level� but envisaged a phased approach, and (ii) the project supported the implementation of the National Strategy for Information Society and the European Union programs related to information society and e-Government service for CIS as one of one of the 20 key e-government services envisaged by the strategy. The project continued to finance the original activities except for internet services. Cancellation: At the request of the Ministry of Public Finance (MPF), the uncommitted amount of US$19,595,436.45 was cancelled from the loan in the restructuring approved by Bank management in February 2013 (see Annex 1.c). It included the amounts allocated for the Extended TA for Communities for the sub-component 1.A included in the restructuring approved in August 2011 (US$4.2 million, see (i)-(iii) under “Extension of the closing date�) and the original and reallocated amounts for the Integrated System for the Issuance of Civil Information, CIS (US$15.1 million) under the sub-component 2.B. The request was motivated by the difficulties in 2010-2012 to provide the adequate Government budget allocation for their implementation (see section 1.7 Project Financing). The request was also motivated by the advanced discussions with the European Commission to use European Structural Funds for implementing the CIS. Project financing: With the cancellation approved by Bank management in February 2013, the loan financing was reduced from US$60 million to US$40.4 million and the Borrower contribution from US$10 million to US$8.9 million (see Annex 1). The main cause leading to cancellation was the insufficient state budget allocation for 2011 and 2012 to pre-finance the loan eligible expenditures and to allow new commitments (see Section 2.2.) in the context of financial crisis. 8 Disbursement arrangements: The August 2011 restructuring also reallocated all the uncommitted loan amounts from the existing expenditure categories into one new single disbursement expenditure category financed 100% from the loan proceeds. This was motivated by the need for an increased flexibility for implementing the project and the exhaustion of the Government contribution. All contracts under implementation continued to be financed under the category of expenditure valid at the time of contract signing. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Soundness of the background analysis. Lessons learnt from previous sector projects were adequately reflected in project design and positively affected the implementation and project outcomes, mainly: (i) the successful Bank experience in working with communities was reflected in project design through demand driven process for establishing the LCeNs and the decentralization of LCeN resources and responsibilities to the beneficiary local administration; (ii) the path taken by many countries in their efforts to develop the information society was reflected by the approach employed by the project to achieve the development objective in three stages (access, use, benefits); (iii) the efficiency of balancing of needs of the communities (bottom-up) and the objectives of the Government (top-down), while ensuring the ownership and commitment and local level was reflected in designing the competitive selection process of the project beneficiary communities among the target group (by including the competition) and in involving local civil servants (teachers, librarians, LCeN managers, public administration clerks), private sector and civil society representatives to be involved in the creation of the content and architecture of the LCeNs; (iv) the use of Banks analytical work and best practices in emphasizing the need for coordination in the implementation of the project on both Bank side (different departments working together) and from the Government’s side (coordination of a number of institutions); (v) the emphasis from the start on the need to focus on sustainability of the LCeNs, and forging their partnerships with private sector in a way that meets the needs of community and generate greater knowledge, technical and financial support in the long run. Assessment of project design. The project was well designed to assist Romania to meet its commitment under the EU programs related to information society for all and implement its own National Strategy for New Economy and Implementation of the Information Society, through a coherent set of interventions aiming to: (i) increase access to ICT and improve digital literacy in knowledge disadvantage communities, (ii) promote e-government services and (iii) promote e-commerce and innovation support for private sector (small and medium enterprises). Alternative approaches were duly considered and clear reasons for rejection were given to both technical solutions (such as a low technology solution with a single node versus complex multi - purpose node LCeNs) as well as to funding and recovery models for the LCeNs (such as a wholly private sector model, wholly public or mixed public-private with a strong public good component). The project design was based on previous work8 on knowledge and digital divide in Romania and on comprehensive sociological studies carried out by the Bank team during 2004- 2005 during project preparation. Knowledge-typologies (K-Typologies) of the communities in Romania were thus defined which proved to be essential for targeting the most knowledge- disadvantaged communities for assistance under the project. In addition, the design was tested9 in 8 Financed from the WB Knowledge Innovation Fund 9 Financed from the Project Advance Facility 9 nine communities and lessons learnt were incorporated in the scale up selection process. The PDO was clear, ambitious and important for the country and the Bank (see Section 1.1) and led to the reduction or even elimination (see Section 3.2) of the digital divide between the communities targeted and the rest of the country, as envisaged by the Information Society Strategy. The three stage approach (access-use-benefits) was appropriate; however the challenge was in achieving the very ambitious targets set for the PDO indicators in the original life time of five years. The components were relevant for achieving the PDO, and technical assistance activities were envisaged to address the knowledge needs and coordination challenges given the technical complexity the interventions, the high number of organizations involved at central and local level and their geographic dispersion, the novelty of the project in Romania, and the limited capacity for implementation, particularly at community level. Adequacy of Government commitment and stakeholders’ involvement. Project preparation oversight was ensured by a Steering Committee led by a strong Minister of MCIS – as National Coordinator for the implementation of the Information Society Strategy; and consisted of high level representatives (at Secretary State level) from relevant ministries (MERYS, MAI, NASME and MCRA). This commitment allowed the proper anchoring and alignment of the project objectives and interventions in the relevant country strategies and policies. In addition, the 255 Local administrations, transparently selected as beneficiaries of the project, demonstrated significant level of ownership (such as assumed obligations for LCeNs governance, management, staffing and sustainability) that proved critical during implementation. Assessment of risks. Risks were generally adequately identified and mitigated both at the level of PDOs as well as to component results, including: (i) project’s complexity, interdependence among components and multi-sector nature; (ii) poor understanding and lack of support from communities and lack of stakeholders; (iii) timing of LCeNs becoming fully operational; (iv) access to ICT and improved Digital Literacy; (v) interest of communities in supporting LCeNs; (vi) e-government institutional capacity; (vii) promotion of e-commerce and innovation support to MSME; and (viii) Project management. By the project’s Mid-term review, the risks were reassessed and additional risk mitigation measures were implemented in the project (see 1.7). An important risk factor was the impact of the Eurozone financial crisis that affected negatively the budget of the Government of Romania (see Section 2.2). Even though this risk was not foreseen in the projects’ design, the Bank team raised constantly the necessity of proper budgetary allocations from the side of the MPF on project level. No Quality at Entry Assessment (QEA) of the project was carried out for this project. 2.2 Implementation Factors outside the control of government or implementation agencies:  The policy commitments derived from Romania’s accession to the European Union in 2007 have determined important modifications in the project context since appraisal, mainly with positive impact towards achieving the development objective and ensuring the sustainability of interventions. Thus, the EU’s “Renewed Lisbon agenda strategy for growth and jobs� followed by the EU 2020 Strategy (and related i2010 and Digital Agenda pillars) reinforced the relevance of the project development objective and increased the need for appropriating the relevant EU policy frameworks at national level. In addition, the availability of European Structural Funds after accession diversified the country’s choices to finance development priorities, with a clear option to 10 maximize the funding from EU grant sources. The project was affected in different ways: (i) the second call for the Grant Facility for SMEs was cancelled (as part of the 2011 restructuring) in view of the EUR 151 million available for similar objectives. Savings obtained were reallocated for the scaled up CIS under subcomponent 2B; (ii) the technical assistance for communities included, as part of the LCeN sustainability interventions, value added services aiming at absorbing EU funds for local economic development objectives. This resulted in the attraction by project communities of about EUR 250 million with a positive impact on achieving the level of perceived benefits targeted by the PDO objective and increasing the sustainability of LCeNs; (iii) the online business services was implemented country wide with European Funds, and (iv) the implementation of the CIS, cancelled as part of the 2013 restructuring and in view of project closure, is financed with EU funding and procurement process is expected to be launched in August 2013.  The global financial crises affected Romania and forced the authorities to embark on a severe austerity program, particularly after 2009. In this context the Government implemented drastic cost reduction measures that affected the pace of implementation and also led to cancellation of project activities (see section below). Factors generally subject to Government control:  The application, starting January 1, 2009 of the mechanisms of the Public Debt Management Law to the whole World Bank investment lending portfolio (including to the KEP) required that the loan financed eligible expenditures be pre-financed out of state budget allocations to the line Ministry and later to be reimbursed from the Bank’s loan in accordance with the internal procedures of the Government. Special Accounts were closed and direct payments out of the loan proceeds were no longer possible. While this mechanism was beneficial for moving towards country systems, it increased the risks for project delays in case of inappropriate levels for advanced financing to cover loan and counterpart funding needs.  The budgetary constraints imposed by the Government, in response to the financial crisis, impacted negatively the smooth implementation of the project. The low level of commitments and disbursements in 2010-11 and the cancellation of loan funds in 2013 were mainly caused by the insufficient project budget allocated to the MCSI through the State Budget law for 2011 and 201210. Thus, the budgetary allocations for 2011 and 2012 were sufficient only for the disbursement of the amounts committed under the existing contracts of the KEP and did not allow new commitments and disbursements to complete the last activities, despite the readiness of the bidding documents and although these activities were planned to be financed 100% from the loan (see “Cancellations� in Section 1.7). The World Bank mission discussed with the Client the details of the restructuring request (that included also a different technical solution for the CIS), and was reassured by the Ministry of Public finance that the needed budget would be made 10 By the time of the project second restructuring (August 2011) the Implementation Progress was downgraded to Moderately Satisfactory given the slowdown of the project activities mainly due to insufficient budget allocation for both loan funds withdrawal and counterpart funding (rated Unsatisfactory). The project Implementation was rated Unsatisfactory by the time of third restructuring (February 2013), due to the same reason. The affected components were: (1 A) the Extended Technical Assistance for Communities (US$4.2 million) and (2B) the Integrated System for the Issuance of Civil Information, CIS (US$15.1 million). 11 available during the September 2011 revisions and that the budget for the year 2012 would be appropriate for the project’s needs. The budget was not provided. This affected negatively the project implementation and caused cancellation of the activity/subcomponent 2.B (CIS). In addition, as part of the cost reduction measures, the Government adopted in 2010 various legislative acts that affected the status and reduced the level of salaries of the PMU staff. As result, all core PMU staff resigned, except the PMU Director. This situation created a serious short term implementation capacity problem. This issue was resolved by the end on 2010 by filling in the core positions (procurement, financial management, etc.) and relying more on the consulting services for ensuring the facilitation at local level.  The Steering Committee functioning was affected by the numerous changes in the Government and affected the pace of project implementation. Since project appraisal, nine Ministers of Communication and Information Society changed function and the numerous changes in the composition and the leadership of the Steering Committee affected the pace of project implementation. Delays in decision-making occurred given the time needed by the newly appointed persons to understand the project. However, the relevance of the project objective and design were not altered by these changes.  The 2011 Decision of the Supreme Council of Defense 11 on ICT systems interoperatibility within the state required that all local administration offices were to be connected into an intranet network owned and managed by the state through the Special Telecommunication Service. This decision affected the approach and technical solution for the CIS. As result, and despite the advanced stage, the bidding process for procuring the CIS was stopped in February 2011. The implementation of the new technical solution that observed the requirements of the Supreme Council of Defense’s decision necessitated the project restructured in August 2011 (Sections 1.6-7). Factors generally subject to implementing agency control: The activity of the Steering Committee and of the Project Implementation Unit was affected by the factors beyond their reach as described above, and foremost by the insufficient budget allocation to pre-finance project activities in 2011 and 2012 which led to delays in implementation and cancellation of loan funds (2013 restructuring). However, the PMU, and in spite of the temporary shortages and of cross sectorial and multi-level coordination challenges, was effective and instrumental in the implementation of the project. The PMU management and staff were of high professional quality and dedicated to project objectives. The technical assistance was effectively used and monitoring and evaluation systems were put in place and effectively used. Mid Term Review (MTR). The MTR took place in October 2008 and provided a series of critical actions necessary for ensuring the achievement in a sustainable way of the development objective and the need for urgent decisions that led to project restructuring. Such recommendations addressed issues aiming to anchor the project interventions in national policies (national strategy for Information society, the Lifelong Learning and Education strategy, and Policy ICT in Education) and e-government programs (Online registration of businesses), to 11 Organism under Romania Presidency and accountable to Parliament 12 increase the sustainability of the LCeNs, to use M&E to benchmarks project achievements to national and European strategic documents, to improve project coordination and management oversight, to ensure appropriate budget allocation, etc. Project changes/restructuring. The project was restructured three times to resolve implementation delays or adjust to changed project circumstances: June 2010, August 2011 and February 2013. The restructurings were necessitated by the need to support the original project activities with additional sustainability measures as identified by Mid-term review, the need to observe the decision of the Supreme Council of Defense that delayed the implementation and changed technical solution regarding the CIS system under the sub-component 2.B, and the availability of EU funding (see Sections 1.6 and 1.7). In terms of effectiveness of restructurings on improving implementation and outcomes, the project extension and the implementation of the additional sustainability measures as to the TA for communities and TA for schools positively influenced the achievement of the PDO and increased the sustainability of the project. The CIS e- government service planned under the project would have increased the availability of e- government services, but its uptake and derived benefits at the level of population, as reduced time and cost of interaction with the Government, would not have been observable within the project life. Therefore, the cancellation of the CIS e-government service activity under Part B did not impact the achievement of the PDO which relies on actual project achievements. After a careful review, the Bank did not agree with the Government request for a second extension to allow the implementation of the CIS given the availability of EU funds to implement it. At the time of restructurings, the loan disbursement status was as follows: US$26.4 million (2010), US$32.8 million (2011) and US$ 37.5 million (2013). 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization M&E design: The original result framework included a fair amount of indicators: two PDO indicators and seven component indicators. Data was obtainable partially from planned surveys or implementation monitoring. Most component indicators were reasonable measures of outputs (grants, MSMEs, SEPs and FAs participating in e-Store activities) or of progress towards LCeNs sustainability objective (in line with development plans), although some indicators measuring use of the ICT were harder to measure (number of trainees who acquired basic literacy skills or used new skills in the classrooms, or improvement in time and accuracy for individuals to complete their transactions through e-government service – for which no base line was available). The Mid-term review stressed on the need to anchor the project results framework in the updated European frameworks and National Strategy for Information Society by using the i2010 indicators and targets as benchmarks for the KE project, as initially envisaged by the PAD. Thus, two PDO sub-indicators were introduced to better explicit and benchmark PDO 1 (which measured satisfaction from the utilization of LCeNs) by proving information on the percentage of population regularly using the internet12 (measuring improved of ICT use) and percentage population using e-government services13 (measuring improved quality of ICT use). The benchmark allowed to better measure the project contribution to achieving the objective set 12 Percentage of relevant population (16-74 years) who accessed internet on average at least once a week, within the last three months before the survey 13 Percentage of relevant population (16-74 years) using the internet for obtaining services from public authorities websites, for downloading official forms, for sending filled in forms within the last 12 months before survey for private purposes. 13 by the i2010 agenda of halving in 2011 the distance (as of 2005) between the average ICT use of EU population and the average ICT use of Romania population, in addition to measuring the achievement of the e-inclusion objective of reducing/eliminating the digital divide between disadvantaged communities and national averages in Romania. Originally there was no separate PDO indicator for the e-government services in the original PAD (as a contribution of component 2); use and satisfaction of e-government services were included as part of measuring the LCeNs and did not differentiate on the type or coverage (local or national) of e-government service. The new PDO indicator introduced in the August 2011 restructuring to reflect the level of sophistication and scaled up approach of the G2C service to be implemented by the project was considered appropriate as it is used by the EC to measure and benchmark the progress of implementing i2020 G2C/G2B services across the EU member states. In addition, six component indicators were appropriately introduced to measure the efficiency, accuracy and integrity of the system to be implemented under the revised subcomponent 2B. This PDO indicator and the six subcomponent indicators were dropped in February 2013 restructuring to reflect the cancellation the procurement of the Integrated System for the Issuance of Civil Information Status under the subcomponent 2.B (Sections 1.6-1.7). The project Mid-term review also stressed on the need to better capture the project activities for the integration of the ICT in schools (under subcomponent 1C) and a new component indicator was introduced. This was a composite indicator of ten sub-indicators that capture the multiple dimensions/outputs of project activities at school level and at national level (see Annex 2, SubComponent 2C). The collection of the data was conducted by the MERYS and was based on self-assessments of the schools participating in the project. M&E implementation: The implementation of the M&E was fully in place by 2008, after the establishment of the 255 LCeNs. Given the large number of communities to be covered comprehensive data was collected annually using surveys. Quarterly reports (covering 85 communities at a time) and in-depth case studies (covering 16 communities at a time) were prepared based on field visits on a rotation basis. In 2008 the MERYS introduced the implementation of self-assessment questionnaires to the 516 schools participating in the project. Data was collected annually. In addition, a separate monitoring and evaluation system was implemented to measure the progress and impact of the activities under subcomponent 3B (Grant Facility). Finally, to measure the overall project impact, by end 2012 Household and Institutional Surveys were implemented in a representative sample 14 of 57 communities out of the 229 communities that remained in the project. Primary data were collected both at the micro (individual, household) level and meso (community) levels through quantitative and qualitative research techniques. In addition, for counterfactual evidence, the household survey was applied in a sample of 20 rural communities that were not project beneficiaries, but that displayed similar initial conditions as the communities included in the project. M&E utilization: The project’s implementation benefitted from the close monitoring and evaluation, as reflected in all Aide Memoires and ISRs and project documentation. Collected data was used to better define the interventions and allocate resources (as for example the TA for communities and the TA for schools), to inform decisions making at both local and central level on shortcomings and/or achievements and trigger decisions such as reallocations, cancellations, etc. The data collected from the Household and Institutional Surveys were used to assess the project overall impact, and in particular for the achievement of the PDO. In addition, the data was 14 95% confidence level and 1.9% margin error 14 used to share good practices in Romania and to report on achievements relevant to European agendas. 2.4 Safeguard and Fiduciary Compliance Environmental Assessment Category (EA). Not Required. Fiduciary compliance Procurement. Overall rating for procurement throughout the project life is Satisfactory. At all times the project had procurement plans that were appropriately monitored and used for decision making. This allowed proper coordination among the different activities, in particular at local level. However, during the period 2010-2012 the overall implementation progress was undermined mainly by the insufficient budget allocations from the MPF and in spite of the bidding documents readiness, as no new procurement could be launched (see also Section 2.2). Financial Management. Overall rating for financial management is Satisfactory. The rating for FM was satisfactory throughout the project life and this included project accounting and reporting arrangements, staffing, internal control procedures, planning budgets and internal and external audits. The project was in compliance with its annual audits and quarterly reporting covenants. The audits were clean, with no internal control issues, and the final audit covering the year 2012 and the final period ended June 17, 2013 was received and accepted in early July 2013. The assessment on “counterparts funding� was downgraded to moderately satisfactory in 2011 and to unsatisfactory in 2012 due to insufficient budget allocation to pre-finance eligible expenditures (for both loan and counterpart contribution - see also Section 2.2). Following the 2013 restructuring and the sufficient budget allocation for covering the remaining project payments until the closing date and subsequent four months graced period, the rating was upgraded to satisfactory. 2.5 Post-completion Operation/Next Phase The e-government service for Civil Information Status (CIS), dropped from the project during February 2013 restructuring, is planned to be implemented nation-wide with EU grants. The financing agreement was approved and the procurement (ICB) will be launched in August 2013 through the e-procurement RO Portal. The implementation of the system is envisaged to take two years. The grant facility for SMEs, discontinued from the project as part of the August 2012 restructuring, continues with financing from the EU grants in total value of EUR 151 million. So far EUR 12 million has been contracted and about 650 projects are in different phases of evaluation. A similar program is envisaged to be financed during the next EU financial program 2014-2020. The World Bank is further involved by providing technical assistance to MCIS for developing the Digital Agenda strategy for the period 2014-2020. In particular, following the experience gained during the development of the eGovernment services and the Bank’s global knowledge assistance was requested for addressing interoperatibility issues. The assistance is provided through Regional Technical assistance for Smart specialization. 15 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Relevance of objectives The PDO is highly relevant to the current country development priorities. The project is fully supported the National Reform Strategy and the European EU 2020 Strategy (and its related Digital Agenda flagship initiative). The project’s indicators and measurements are consistent with key performance indicators used by the EC to benchmark the progress of each EU member states towards the European Digital Agenda goals and targets. The PDO is also aligned to the current Country Partnership Strategy FY09-FY13 and Progress Report that supports Romania's convergence with the EU, in particular to increasing the participation of the disadvantage communities in the knowledge based society and towards implementing the goals of the EU 2020 strategy. Relevance of design and implementation The project design is highly relevant and is rated highly satisfactory. The three stage approach for project components’ activities (access-use-benefits) remains highly relevant for the country’s development challenges and for achieving the PDO. Firstly, the project provided access and connectivity through the establishment of LCeNs with access nodes in schools, public administration, public library and public internet access points (PAPIs) in the selected communities under Component 1. Secondly, the project provided technical assistance to increase the skills and services for accelerated use of ICT both at local and national level - under component 1 and 3. Thirdly, the project supported improvement of the quality of ICT use through its integration in the education process, use of government e-services and online transactions by businesses and citizen, while ensuring sustainability of project intervention – under Components 1, 2 and 3. Components’ activities were logically sequenced and coordinated at central and local level. The combination of multi-sectoral interventions envisaged by the three components was critical for achieving the impact on the ground, as evidenced by the project impact assessment (see also Section 3.2). In addition, the implementation of the project by a centralized PMU within the MCIS as designed by the project, as opposed to multiple PMUs located in the different line ministries (an alternative rejected during project preparation) was key for overcoming coordination challenges derived from the cross sectorial and multi-level (local-central) nature of intervention. The availability of EU funding after Romania’s accession into EU in 2007 was used opportunistically during project implementation and led to adaptive project restructurings that included reallocations and cancellations. First, the second call under the Grant facility was cancelled given the parallel financing for same objectives, and project funds were reallocated; second, in view of the advanced discussions with the EC for securing funds for the CIS implementation and due to the previous difficulties in getting the necessary budget allocation, the Bank did not concur with the request of the MPF to extend the closing date by 22 months to complete this activity. The design was appropriately adjusted to reflect what the project financed. Third, and in the context of the legislative changes that eliminated the local authorities as intermediaries in the businesses registration process, the utilization of the system developed by the project was stopped; however the availability and usage of the online registration by the project target group (individuals and family association) was ensured following the implementation at national scale of a special module for online registration dedicated to this 16 target group, based on the knowledge created under the project. The module is part of the overall online registration system implemented by the NTR with EU funding. 3.2 Achievement of Project Development Objectives The project contributed to accelerating the participation of the targeted 255 (knowledge disadvantaged communities (208 communes and 47 small cities or 44 % of the total knowledge disadvantaged communities) in the knowledge-based society and economy in Romania to a high extent, as demonstrated by the outcome indicators. Specifically, in the targeted disadvantaged communities, LCeNs evolved into a daily tool for education, business and public communications with the Government, with increasing private management and financial self-sustainability. The two outcome indicators measuring the progress towards the PDO have been met and exceeded, as follows: Firstly, 43 % of relevant population (age 16-74 years), higher than the target of 40%, is satisfied with the project results. As demonstrated by the results of the Household survey implemented in the KEP communities, the probability of project derived benefits is increased significantly by the value-added services provided by the PAPIs as a central node of the LCeNs, besides the Schools, Library and Local Public Administration nodes. These value added services included assistance for accessing on-line services in the interaction with the local and central Government, e-commerce and marketing assistance to entrepreneurs, employment related services, health and cultural related services. The CIS e-government service planned under the project would have increased the availability of e-government services, but its uptake and derived benefits at the level of population, as reduced time and cost of interaction with the Government, would not be observable within the project life. Therefore the cancellation of the CIS e- government service activity under the subcomponent 2.B does not impact the current PDO rating which relies on actual project achievements. Secondly, 30 % of the LCeNs operate in a sustainable manner having concluded partnerships with different other public or private organizations, higher than the targeted 20%. In addition 51% of the communities have projects pending signing. As result of these partnerships more than EUR 250 million was attracted in the communities, including for the development of the LCeNs. In addition, the Institutional Survey with the KEP communities’ mayors elected after 2012 provide additional information about the future financial sustainability of the LCeNs as over 89 % of them consider that LCeNs’ Internet and operational costs will be covered and PAPI will continue operate. The project enabled the Borrower to implement the National Strategy for the New Economy and Implementation of the Information Society, and the European Union programs related to the information society for all, as proved by the indicators measuring the reduction of digital divide as the level of access and use and meaningful use of ICT against the national and European targets. Thus, in terms of access in the targeted communities, (i) 43% of the relevant population with no access to Internet before the project has now access to Internet from any public point (of which 38 % of population use PAPI as access public point), higher than 38% in the control groups (non KEP communities with similar initial conditions) and (ii) due to project externalities 50.5% of the relevant population with no access to Internet before the project has now access to Internet from home, higher than 38% in the control groups (non KEP communities with similar initial conditions). In respect to regularity of use, (i) 41 % of relevant population uses regularly the Internet (measured as percentage of population that used every day or almost every day or at least once a week within the last 3 months before the survey), higher than the control group (35%) and almost equal with the 17 national average of 43% and (ii) 12 % of relevant population uses “meaningfully the ICT (measured as percentage of population that used e-Government services, for obtaining information from the public authorities websites, for downloading official forms, for sending filled in forms) higher than 3 percent in control group and 12 % the national average. These indicators are measured at the level of relevant population, namely 16-74 years. In addition, nearly all children (3-15 years) in KEP communities have access to ICT either at home or at schools and other public points and 88% of them ever used a PC and 73 % ever used Internet, as compared to 63% and 57% respectively in control non-KEP communities. The benefits are even higher in comparison with control group in terms of regular use of Internet as 73% of children use ICT in schools (as compared to 47%) and 71% benefit of regular ICT supported classes (as compared to 38%). The project contributed to achieving enhanced individual informational capabilities of people living KEP communities realized through (i) increased availability to ICT equipment and Internet; (ii) realized functions through increased usage of ICT; (ii) increased meaningful use of ICT in informational, social, economic, vertical and horizontal communication and cultural areas (equaling or outperforming national averages and targets - see Annex 5), in line with the National strategy for information society. The project also contributed to achieving enhanced social capabilities at community level in terms of informational, organizational, social and economic development, political participation and cultural identity (see Annex 5). The Project Impact chain and linkages between outcomes, outputs and activities, used also as conceptual framework for project evaluation, are summarized in the figure below (see also Annex 2). The main Outputs per components are the following:  The project delivered all the outputs under Component 1 to improve access to and use of ICT in the knowledge disadvantaged communities and achieved the component targets to a high extent. 229 LCeNs are operating satisfactorily in line with the satisfactory 3-year LCeN sustainability plan. The datasheet demonstrates increased integration of ICT in education, increased number and diversified LCeNs services and improved digital competencies above the targets (see Annexes 2 and 5 and datasheet). The last activity under this component, the Extended Technical Assistance for Communities (est. US$4.5 million), was not launched given the budget limitations (see Section 1.7). It would have aimed at consolidating the good results from the previous project interventions and disseminated the good practices in other communities. It did not negatively affect the achievement of the targets. It would have supported and exceeded the existing PDO indicators.  Based on the approved restructuring, the project delivered the agreed outputs under Component 2 to develop and promote e-government services and met the revised targets (Sections 1.7 and 2.2). Online registration of entrepreneurs and family associations, targeted by the project, was implemented nation-wide as a result of the knowledge achieved under the project through the addition of a special software module to register these categories of businesses targeted by the project. The utilization of the online system developed under the project was stopped in 2011 following the adoption of a new legislative framework governing the registration of entrepreneurs and family associations that excluded the role of local authorities as facilitators in the registration process. Hence, the indicators that measured improved efficiency of local authorities as facilitators were dropped as irrelevant for the new registration process. The revised design excluded the implementation and deployment of the Integrated System for the Issuance of Civil Information (CIS) (est. US15 million) from the 18 project given the budget limitations. The bid documents prepared under the project are used for procuring the system with funding from EU.  Based on the approved restructuring, the project delivered the outputs under Component 3 to establish the e-Portal (e-comunitate) for promotion of e-commerce, business networking and community development and attracted cumulative visits above the target (1104995 vs. 500,000) and provided grants to 41 SMEs (the project cancelled the second call of grants because of the availability of EU funding for SMEs to adopt e-business solutions). The below chart describes the result chain of the KEP from outputs to outcomes for the implementation cycle from developing access to diversifying use and to increasing benefits: 19 ACCESS USE BENEFITS Outputs Access Outputs Outputs Outcomes • 255 (or 44 % of k- Simple Use of ICT Meaningful Use of 43% of relevant deprived ICT population (vs. 40 • 510 LCeN staff communities) trained in basic • 90% of LCeNs % target) is selected in the digital (229) developed satisfied with project); competencies; and implemented project results • 255 LCeNs fully satisfactorily 3- given: • 510 LCeN staff year sustainability established, linked trained in project • Enhanced to internet ( over business plans individual management (1B); 100 km fiber (1A); informational optic) and nodes • Increased number capabilities (255 PAPIs, 502 • 255 LCeN staff and diversified trained in network realized through schools, 255 value added increased mayories, 255 administration services provided availability to ICT public libraries) • 255 LCeN staff through LCeNs equipment and equiped with PCs, trained in local (1B); Internet ; realized ICT audio/video economic • Increased level of functions through equipment, development ICT integration in increased usage of furniture, etc (1A); education (93 % ICT; increased (1A): • over 200 persons of target ), meaningful use of • 510 LCeN staff from KEP measured as ICT in selected and hired communities aggregation of 10 informational, by local trained for digital breakdown social, economic, administration competencies indicators) (1C); vertical and (1A). (1A); • 75% of teachers horizontal • Call Center • over 6400 used new skills in communication established for participants in teaching (1C); and cultural areas online assistance 140 info-seminars • 75% of people (equalling or (1A). for schools (1C). trained in ICT outperforming used new skills in national averages work (1B) ; and targets - see • Over 1.1 million also Annex 5), in visits in the line with the ePortal (against National strategy 0.5 million target) for information and more than society 9,066 users • Enhanced social registered (3A); capabilities at • 41 SMEs community level implemented e- in terms of business solutions informational, (3B); organizational, social and • Available Online economic registration of development, entrepreneurs and political family participation and associations (2A, cultural identity followed by (see also Annex nation wide 2). implementation based on project built knowledge and EU funding ) • Technical specification available for the nation wide implementation of eGovernment services for civil status ( 2B and further implemented with EU funding). 20 In recognition of its achievements the project received medals and awards from prestigious international and national institutions, such as: Certification of Good Practice at the �European Public Sector Awards� (EPSA) within the European Institution of Public Administration (Brussels 2011), Laureate of Computer World Honors Program (Washington 2011), 1st Prize at the National Competition of Best Practices in Public Administration (Romania), finalist at the International Project Excellence Awards 2010 organized by the International Project Management Association (IPMA), Medal at the e-Inclusion Competition organized by the European Commission, at the section “Geographic Inclusion� section (Vienna, 2008) and "Project with the best informational content" at the Romanian IT&C Awards (Bucharest, 2006). 3.3 Efficiency The efficiency of the project in achieving the original objectives is substantial. The benefits brought out by the project outweigh the project costs. Besides the surplus generated from the value beneficiaries place on LCeNs basic and value added services (the project activities that are difficult to be quantified in monetary terms), the project helped attract additional funding of more than EUR 250 million in the project communities, increasing the sustainability of the LCeNs. In addition, the project activities cancelled from loan financing were further financed by EU grants funding (grant facility for SMEs and CIS implementation) or by the local budgets of the project communities (salaries of LCeN staff). The funding and recovery strategy implemented by the LCeNs was in line with the main features of the “adoption� model� envisaged in the PAD, by which the Government contribution was to decline over time till its elimination by the end of the project, while commercial and donor funding sources were to be identified and attracted. The decline of the project contribution was more accelerated than initially envisaged as a large part of the LCeNs operating costs (salaries of the LCeNs staff) that was designed to be financed by the loan for the whole duration of the project, was in fact funded from the local budgets of the targeted communities (salaries of the LCeN staff). In addition, other funding strategies were implemented (EU or other donors 15 funded projects, private-public partnerships, fees for business support services and training) to increase the operating revenues of the LCeNs. The declined loan funding mechanism implemented by the project and the stress on delivering diversified LCeNs services helped ensuring the financial break-even and sustainability of LCeNs. The project extension of 18 months beyond the original closing date made possible the implementation of additional risk mitigation measures - the intensified technical assistance to communities and to schools - which increased the impact and the sustainability of the 15 Bill and Melinda Gates foundation 21 project. In addition, the project externalities are significant in terms of Internet penetration in the target communities. The downside in terms of efficiency is related to the delays in the implementation of the Integrated System for the Issuance of Civil Information (CIS), mostly due to the insufficient budget allocation in 2011 and 2012, which finally led to the cancellation of the relevant portion of the loan. However, the activities done under the project help the authorities develop the bidding documentation for implementing the system with EU grant funding. Thus, the cancellation of the relevant portion of the loan, as opposed to the decision to extend the project closure to finalize the implementation of CIS with loan financing, allowed for a least cost alternative available at present, given the availability of EU grant funding. 3.4 Justification of Overall Outcome Rating Rating: Satisfactory The overall outcome rating is satisfactory. The project achieved its highly relevant PDO in accelerating the participation of knowledge disadvantaged communities in the knowledge based society and economy in Romania to a high extent. The household and Institutional survey conducted by the World Bank 2012 in the project communities (KEP) shows important development impact, equaling or exceeding national and rural area averages along most dimensions of the ICT impact chain.16 Efficiency remains substantial given the benefits brought out by the project in spite of implementation delays with the CIS system. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development The project included insightful needs assessments that helped selecting the project beneficiary communities. The selection of the rural communities was based on a grid that took into account several indicators, as follows: (i) the level of access to population to internet and information (telephone and TV); (ii) the knowledge deprivation score 17 (KDP); (iii) the dimension of the target groups; (iv) local institutions needs in terms of information; and (v) community participation and sustainability plans. As a result of the selection, the KEP beneficiaries were communes and small cities displaying the most deprived level both in economic and knowledge terms. At the outset KEP covered over 1.09 million persons from 255 localities (2002 Census). Due to the general decline of population and the drop out of 26 localities, KEP coverage at closure diminished to 1.05 million persons in 229 communities. The project reached all major groups of population, irrespective of gender, age, ethnicity, and social strata. Although vulnerable groups, such as women, Roma people or elderly were not explicitly targeted by the project, they benefited as well from the project. The digital use and regular use divide is present between rural and urban within the KEP area, as well as between KEP and non-KEP rural communities and are both statistically significant. Non-KEP communities with similar initial conditions as KEP communities have poorer levels of access and 16 Please refer to annex 5 – Beneficiary survey results 17 Knowledge deprivation score represents a factor score of KEDUC index, TV and phone subscriptions at 100 households, distance to the nearest city, share of commuters in active population, share of 65 and more age population; KEDUC index is a factor score of the community stock of education, share of population of age 10+ with post- secondary education, share of post compulsory education students in 15-29 age category and post mandatory school unit in locality. 22 ICT skills than KEP communities. The use divide is high in KEP area, nevertheless smaller than in the non-KEP communities. Thus, the regular use of a computer and/or Internet is correlated with many factors, including age, gender, human capital, employment status, household's livelihood strategies, and community capitals. In KEP communities the regular use of a computer is significantly more frequent among: younger people compared with elderly,18 men compared with women19 as well as Romanian and Hungarian ethnics compared with Roma people.20 A particular case is the digital divide between Roma people and the rest of the population. KEP has not closed the gap between Roma and the Romanians regarding access to and usage of ICT, but as result of the project Roma in KEP communities are better off than Roma in control group communities, which means that they are better than they would be if KEP would not have been implemented. Roma represent 3.3% of the population covered by KEP (more precisely, 1.2% of KEP small urban population and 4.9% of KEP rural population),21 which is about 33 thousands persons.22 Roma adults aged 16 to 74 years have an ICT usage much lower than the other ethnic groups: (i) only 53% of Roma have ever used a mobile phone compared with 83% of Romanians; (ii) 26% of Roma have ever used PC compared with 50% of Romanians; (iii) only 21% of Roma use regularly a PC compared with 43% of Romanians; (iv) 12% of Roma have ever used Internet compared with 47% of Romanians; (v) only 12% of Roma use regularly Internet compared with 42% of Romanians. The high usage gap between Roma and the other ethnic groups is strongly associated with a lower access to ICT equipment and infrastructure. Roma are usually grouped in poor and remote villages (or neighborhoods), hence they are overrepresented among the population geographically disadvantaged. 23 Most Roma households are large, with many children and little incomes and many face poverty. Consequently, they are overrepresented among people who cannot afford equipment and, particularly, the monthly costs for phone or Internet subscription. Thus, out of the whole Roma population (all ages): 53% persons live in households with one (or more) mobile phones (versus 86% of Romanians), 51% Roma persons live in households with access to PC at home (versus 55% of Romanians), and 25% Roma live in households with access to Internet at home (versus 48% Romanians). (b) Institutional Change/Strengthening The project has a substantial institutional development impact in the targeted communities. The design and implementation of the project followed the ICT impact chain approach by providing access to ICT, use of ICT, and through value added services boosting the meaningful use of ICT for deriving benefits in various areas of life. The project contributed to enhance the 18 In KEP area, the proportion of individuals 16 to 74 years old who use regularly a computer decreases from 76% of 16-29 age category to 49% of 30-39 years, 45% of 40-49 years, 33% of 50-59 years, and only 9% of 60-74 age category. In non-KEP area, this indicator ranges between 63% of young (16-29 years) and 4% of elderly (60-74 years). 19 The proportion of individuals 16 to 74 years old who use regularly a computer is: 46% of men compared with 39% of women, in KEP area; 44% of men and 36% of women, in KEP rural area; 41% of men versus 31% of women, in non- KEP rural area; 47% of men and 43% of women, at the national level (data Eurostat for 2012). 20 In KEP area, the proportion of individuals 16 to 74 years old who use regularly a computer is 43% of Romanians or Hungarians, but only 21% of Roma. However, in the control rural communities virtually no Roma use a PC on a regular basis. 21 In 2012, after the project was adjusted, KEP covered 229 local communities with approximately 1.05 million inhabitants. 22 KEP Roma population is gender balanced, but is much younger compared with the Romanian population: more than half (51%) of them are children 0-15 years, while among the Romanian ethnics this age category represents only 14%. 23 Without easy access (at home or at a place with public access found at less than 2 km or 15 minutes distance walk from home) to PC/ Internet. 23 capabilities of individual and communities for social and economic development, local participation and community empowerment and the capacity of local institutions to provide ICT related value added services. The most important two benefits, as indicated by the surveyed population, are "better schools within communities" and "enhanced level of civilization and openness to world of citizens". In addition, the Ministries involved in the project increased their capacity to implement large and complex project that require intra-Government coordination and involve a large number of communities. (c) Other Unintended Outcomes and Impacts (positive or negative) None (the PDO was fairly broad and covered most of the benefits that resulted from the project although some benefits were not quantified. For example, the project goal was to make LCeNs sustainable and it measured it in terms of % of LCeNs sustainable, but it did not set any targets for the amount attracted. The value of partnerships is significant, perhaps more than anticipated (see Section 3.2), but it can't be said that the outcomes are unintended because these funds will help LCeNs to continue to provide a daily tool for education, business and public communications with the Government, as planned) 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops According to the beneficiaries interviewed as part of the Household and Institutional Survey, the project was largely successful, that is most stakeholder groups attained their major goals and did not experience significant undesirable outcomes24. Actually, the project has been awarded by prestigious national and international organizations (Section 3.2). Following the ICT impact chain: the access to information and ICT has been significantly improved; the basic ICT skills were ensured; the higher Internet proficiency accompanied by guidance and support provided through PAPI services has resulted in meaningful use of Internet, which led to enhanced informational capabilities; and thus the project has had a positive impact on well-being, both at the individual and community level. The multiplier effect of informational capabilities varies from one area of life to another. Positive impact has been highlighted in the informational, social and, especially, in the cultural areas. Less extended impact was observed in the economic (income generating) and vertical communication (eGovernment) areas, however high in relative terms (compared to the national average), particularly taking into consideration that the impact in these areas is limited by broader socio-economic factors found out of the scope of the project. In view of the beneficiaries, the services provided through LCENs (at PAPI) played the most important role in enhancing people's informational capabilities and in increasing the odds of meaningful use of Internet in order to derive benefits in the social, economic, political and cultural areas. PAPIs played the role of an ICT and social intermediary as a rather successful process of technical and social appropriation took place at the local level. The prospective for PAPIs future seems optimistic, according to the mayors of the participating communities. Nonetheless, sustainability remains an issue to be considered. 24 The project carried out an impact assessment of the KEP in two phases, May-June 2012 and January 2013. Read more in Annex 5. 24 4. Assessment of Risk to Development Outcome Rating: Moderate The sustainability of the project impact is significant at the level of the communities assisted by the project. 229 communities (or 90%) of an initial number of 255 communities initially supported by the project, implement measures in line with their approved sustainability plans. 30% of the communities have concluded partnerships and are implementing various development projects. Moreover, 51% of communities have projects pending signing. About 80% of the mayors of these communities who were elected in 2012 consider the LCeNs as financially sustainable after project closure. The interventions made at the school level are currently anchored in national adopted policies. In addition the local authorities took over the operational costs of the LCeNs and the cost of equipment renewal, initially envisaged to be financed by the project. Despite the optimistic view of the beneficiaries, there is a risk regarding the sustainability of the LCeNs and in particular of PAPIs which have been so important in driving forward the program. The risk is however mitigated due to (i) the externalities already created by the project (Internet penetration and use at house level in the targeted communities), (ii) the availability of EU funding over the next seven years (2014-2020) allowing communities to further implement ICT driven projects and implement digital agenda’s objectives and by using PAPI as intermediaries, and (iii) the knowledge and digital competencies built under the project that are nor resting only with LCeN staff. The sustainability of project interventions at central level and knowledge built are also significant. As described in Section 2.5, (i) the Government implements the CIS nation-wide using the European Funds and by involving the same core technical team of the MCSI and MAI; (ii) the online registration of businesses is a core service provided by the National Trade Registry; (iii) new e-business programs are planned for the next financial programming period 2014-2020, built on the knowledge created under the project; and (iv) digital competency is recognized as one of the eight core key competencies at national level and reflected in school curricula, and teaching and learning standards, while teaching lessons developed under the project are accessible nation- wide. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory The Bank’s performance during preparation is rated satisfactory. The project adequately responded to the sector and to country’s demands with the objective to help Romania implement the National Strategy for the New Economy and Information Society as well as EU programs related to information society for all. The project design included a coherent set of interventions to tackle market failures in a very near future. The project was designed to introduce the needed investment, services and TA in a planned phased way and at several points, competitive mechanisms and alternative management and financing models. The project promoted institutional building, increase of capacities and coordination among government institutions. The project was designed to be flexible given the innovative nature of the interventions. It included a learning process through various pilot activities and use of alternative models for LCeNs. 25 (b) Quality of Supervision Rating: Satisfactory The Bank’s performance during implementation is rated satisfactory. The project supervision was adequate (approximately 12 implementation support missions with an average of two missions yearly), and provided the necessary close support throughout implementation. Although the project implementation was very complex and supervision was challenged on many aspects, the Bank responded proactively and in all fronts, by taking the necessary steps to keep the project in track. All three restructuring of the projects were conducted with great caution to address the identified shortcomings or respond to new developments or to new difficulties created by various factors or to strengthen the sustainability of the project results. Attention was paid to preserving project’s consistency with PDOs and ensuring compliance without significantly modifying project outcomes. The team timely revised project risks and introduced risk mitigation measures to ensure project sustainability. The team focused on monitoring the progress based on carefully chosen indicators and invested resources above the average in assessing the impact of the project on the communities and compared to national averages. The fact that the project had one team leader almost from its start was an important factor that contributed to the quality of supervision, consistency of documentations, clarity of messages and efficiency of actions and advice from the Bank’s side. Overall technical quality of the team members was also high. Fiduciary aspects of the project were also conducted in a satisfactory manner; procurement was always assessed satisfactory, despite the technical complexity of the contracts and their size. Financial management was also satisfactory. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory Overall the Bank performance is Satisfactory on both ensuring the quality at entry as well as during supervision. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory The borrower’s performance is rated moderately satisfactory. The project was very complex and among others presented a lot of challenges in terms of coordination amongst implementing institutions. As evidenced by project supervision documents, there were difficulties in terms of activating coordination, communication and strategic leadership mechanisms. However, over the years the coordination among the institutions involved in the project, both at policy and technical level, improved. Protocols of cooperation were concluded between MCSI and MAI and STS for implementing the CIS project and between MCSI, MERYS and the Ministry of Culture for the TA for ICT integration in schools. The frequent changes in the leadership of the involved Ministries and in particular of the Ministry of Communication and Information Society influenced negatively the path of project implementation as they delayed decisions and challenged project ownership. In addition, the inability of the MCSI and MPF to mobilize the necessary budgetary resources during the last two years made it impossible to complete the two last project activities and negatively affected achievement of related outputs. 26 (b) Implementing Agency or Agencies Performance Rating Satisfactory The PMU performed well during the entire project life and in spite of the temporary shortages and of cross sectorial and multi-level coordination challenges, was effective and instrumental in the implementation of the project. The strong PMU director and staff were of high professional quality and dedicated to project objectives. The technical assistance was effectively used and monitoring and evaluation systems were put in place and effectively used. The PMU regularly monitored and evaluated the project activities including by implementing questionnaires and case studies. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory Overall the Borrower Performance is Moderately Satisfactory. The activity of the Steering Committee and of the Project Implementation Unit was affected mostly by the factors beyond their reach as described above, and foremost by the insufficient budget allocation to pre-finance project activities in 2011 and 2012 which led to delays in implementation and cancellation of loan funds (2013 restructuring). However, the PMU, and in spite of the temporary shortages and of cross sectorial and multi-level coordination challenges, was effective and instrumental in the implementation of the project. 6. Lessons Learned The staged approach (access-use–benefits), the in-built flexibility to adjust interventions to specific needs of the communities to increase the uptake of ICT for derived benefits and the close monitoring and adjusting of indicators as the implementation evolves are critical success prerequisites for innovative and complex projects like this. The project provided access and connectivity through the establishment of LCeNs with multiple nodes in schools, public administration, local libraries and PAPIs; secondly the project provided technical assistance to increase the basic digital competencies for using computers and Internet; lastly, the assistance was devoted towards meeting the user needs and perspective, improving the quality of use and reinforcing the trust and confidence to encourage the use and development of e- government services and carry out online transactions, while becoming more aware of security issues. The intensified TA to schools and TA to communities were essential in this respect and the proper use of monitoring allowed for timely adjustments in project focus at local level. This last stage was critical for achieving benefits as demonstrated by the results of the Household and Institutional survey that showed that KEP communities are better off than non-KEP communities from the control group that had initial similar condition, both in terms of regularity of use, but more importantly in terms of meaningful use of ICT. Most of the non-KEP communities benefited from interventions from other Governmental programs or EU funded programs, however they were fragmented, and limited only to providing access to Internet and at most to building digital competencies to allow simple use of ICT. Multi sectorial-interventions are necessary to tackling different communities’ needs however the project design needs be based on appropriate assessments to allow selectivity to ensure the balance between project complexity and optimum impact. The project included insightful needs assessments that helped select beneficiary communities, identify their needs, design the multipurpose node business model for the LCeNs and define the scope of the technical assistance needed to respond to the different stakeholders (teachers, students, clerks, businesses, population at large). Thus the LCeNs’ nodes were selected to be established in schools, Mayor 27 Office, local library and PAPI as the most critical knowledge access points for the communities. Technical assistance was tailored to respond to the specific needs of the stakeholders and project results reconfirmed and reinforced the multipurpose business model for LCeNs. PAPI role needs to be understood in the evolving community context and supported with different instruments to be able to address increasingly more sophisticated community needs. PAPI’s role, as envisaged by the project was of a Broadband universal service25 provider for the selected communities, but also a catalyst for the uptake of ICT and derived benefits. While in initial stages of the project implementation PAPI was established and mostly used for accessing computers and Internet, the increasingly higher internet household penetration (expected externality) and more sophisticated needs of the beneficiaries, determined PAPIs to reconsider their role by focusing on providing value added services (business support service provider - in particular to farmers, knowledge and training center, project management for local economic development, information and tourism center, etc). The project supported this transition by first providing the ICT endowment and training of LCeNs staff to ensure the simple of ICT, and moved towards more customized assistance based on specific local needs (TA for communities) for supporting PAPIs to provide 26 value added services. As revealed by the Household survey, more than 70% of the PAPIs beneficiaries declared to be (very) satisfied with regard to nearly all services provided. Governance and ownership at local level are critical risk factors for LCeNs’ sustainability, that are particularly high in crisis periods, and mitigating measures are needed to achieve sustainable results on the ground. Sustainability of LCeNs was an obsessed concern of both the PMU and Bank’s team, particularly since the project was to address most of the k-disadvantaged communities in Romania with a very low base start-up. Mitigating risks measures were considered from the design phase and additional ones were developed during implementation. Given the lessons learnt from the pilot stage and the quality of the LCeNs development plans prepared by the communities as part of the selection process, the project teams stressed on the need for additional technical assistance to communities for increasing the sustainability of project interventions. Sustainability plans were thus developed based specific knowledge needs and corresponding actions identified in consultative processes with targeted beneficiaries. 90% of the communities initially selected in the project approved the sustainability plans as part of their Local development plans supported by the local budgets and pursuit with their implementation. The project activities, outputs and outcomes for the community, generated strong ownership and resulted in higher counterpart contribution27 than initially envisaged, even during a financial crisis. The fact that most of the new mayors elected after 2012 in the KEP communities support and commit to support financially the sustainability of the LCeNs is an additional proof of that. The project interventions at school level need to be anchored in national policies for sustainable results and results to be shared nation-wide. The project supported both policy 25 "Universal broadband service", refers to Governments obligations to ensure all citizens have access to Internet in white areas. 26 The number of services offered per community varied from 1 to 22, with an average number of 14 services (Kantor, M&E Report). 27 Salaries of LCeNs staff were to be financed from the loan proceeds. 28 changes at the national level as well as intervention at school level. In the absence of a Policy for ICT in Education at the beginning of the project, the interventions at school level were running the risk to be left as pilot attempts. Much attention was given to assist MERYS to develop and implement policies and strategies that provided a comprehensive approach to digital competence development, encompassing national curricula reform that integrate and assess digital competencies within and across subjects, revising teacher ICT competency standards in terms of pedagogy practice and professional development, curriculum and assessment, school organization and development. A multitude of Guides and Books for ICT in Education were prepared and disseminated with other schools via Internet and in hard copy. The difficulties in designing and implementing e-government services should not be underestimated and need to be considered in the broader context of the public administration reform efforts. While the project had an appropriate design by envisaging ICT uptake from both the demand side and supply side, the implementation of the nation-wide e- government services under Component 2 (CIS and online registration of local businesses) posed not only technical difficulties, but also implied the review of the internal businesses processes of the organizations involved that required time to address. The use of the ICT was not for automating existing procedures, but rather to streamline and made more efficient the authorizing environment. The implementation was much delayed in the case of the CIS due to concerns of security of information, but also to changes in the legal and institutional framework. Although the project could not implement the CIS, it built significant knowledge and strengthened the inter- institutional cooperation. These achievements allow for a now smoother implementation with EU funds, which otherwise would not have been possible. The results achieved under the KEP demonstrate the comparative advantage of the Bank in designing and implementing complex and innovative projects. Although an EU accession country at the time of the start of the project, Rumania needed a holistic approach in development of an information society and increasing penetration of ICT even in the most disadvantaged areas of the country, to be able to integrate and compete within the EU. The Bank was able to deliver a project that could serve the sustainable achievement of the country’s objectives and even when there was not possible to complete the activities within the project, it prepared the technical ground for follow up under EU financing. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies The Borrower rated the Performance of the Borrower as Highly Satisfactory based on a Highly Satisfactory rating for the PMU performance and Highly Satisfactory rating for the Government. However, due to major shortcomings, in particular in terms of ensuring the necessary budget for the smooth project implementation that further led to the cancellation of two project activities, the Overall Rating for the Borrower is Moderately satisfactory. (b) Cofinanciers None. (c) Other partners and stakeholders None. 29 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Component 1. Access to ICT in disadvantages 43.0 38.0 88 communities and improved digital literacy Component 2. Development and promotion 8.9 0.8 9 of government e-services Component 3. Promotion of e- commerce and 11.9 3.5 29 innovation support for MSMEs Component 4. Project Evaluation and 6.0 4.8 80 Implementation Total Baseline Cost 69.8 47.1 67 Physical Contingencies 0 Price Contingencies 0.2 Total Project Costs 70.0 67 Front-end fee PPF 1.8 Front-end fee IBRD Total Financing Required 70.0 48.9 70 Note: The actual data is from the restructuring paper approved in February 2013. The project is expected to disburse 100% of the revised loan amount of US$ 40.4 million by the end of the grace period. (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Borrower28 10.0 8.50 85 International Bank for Reconstruction 60.0 40.40 67 and Development 28 Includes only Central Budget contribution 30 (c) Changes of Estimated Project Costs based on Restructuring Papers Total Total Total Total Difference Cost Cost RP1 Cost Cost RP3 from PAD29 2010 RP2 2011 2013 PAD (of which (of which (of which (of which (of which Loan) Loan) Loan) Loan) Loan) Part A. Access to ICT in Knowledge Disadvantaged 43.1 41 43.5 38 -5.1 Communities and Improved Digital Literacy (37) (35) (36) (31.8) (-5.2) Part B. Development and 8.9 8.9 16 0.8 -8.1 Promotion of e-government services (7) (7) (15.8) (0.7) (-6.3) Part C: Promotion of e- 11.9 11.9 3.5 3.5 -8.4 commerce and innovation support for MSMEs (11.3) (11.3) (3.1) (3) (-8.3) 6.1 6 5.2 4.8 -1.3 Part D: Project management (4.7) (4.7) (3.6) (3.4) (-1.3) Refunding Project Preparation 2.2 1.8 1.8 1.8 Facility/Contingencies (2) (1.5) (1.5) (1.5) Total Project Cost 70 70 70 48.9 (-21.1) (of which Loan) (60) (60) (60) (40.4) (-19.6) Notes: Relative to the original allocation per components, the cancellation of US$ 19.6 million reflects the changed described below. 1. About US$ 13.3 million of the cancelled amount represents savings generated throughout the project life. Thus:  Part A: Savings of about US$ 5.5 million representing the Operational Costs for the functioning of the LCeNs (salaries of the LCeNs staff, LCeNs office supplies, LCeNs rent of office accommodations): The operating costs incurred by the LCeNs estimated at US$ 8.5 million and planned to be financed entirely from loan proceeds, were in reality covered jointly from the project funds and from the local budgets of the beneficiary local authorities. The loan financed the internet and telephone costs, depreciation, system upgrades and maintenance. The other costs, such as office rentals, utilities other than internet and salaries of the LCeNs staff, were taken over by the local authorities. This measure followed the lessons learnt from the pilot stage reviews to implement measures to increase the local ownership of the project. In addition this measure addresses other issues, such as: the consistency of the LCeN staff salary and the public pay levels and frameworks; sustainability of project interventions; and difficulties in managing a PMU with more than 500 consultants to cover the 255 communities. The savings were reallocated among other expenditures 29 Does not reflect the Project Preparation Facility and unallocated amounts per Loan Agreement 31 categories within the component 1 (in the 2010 restructuring) and were planned to be used mainly to finance risk mitigation measures identified by mid-term review, such as the intensified technical assistance to communities and to schools. These activities were to be measured under the originally PDO results indicators. New intermediate indicators have been added however to reflect more accurately the activities performed.  Part C: Savings of about US$ 5.8 million from the cancellation of planned calls under the Grant facility: Given the availability of EU grant funds for the adoption of e-business solutions by firms in an amount of about EUR 151 million, the Project Steering Committee decided not to launch the second call under the grant scheme and to finance the program from grant sources. The resulting savings were reallocated to component 2 and target of the relevant intermediate indicator was reduced from 300 grants to 41 (RP 2011).  Part D: the loan allocation was reduced by $1.3 million. This represents the savings from the Project Management unit operational costs. 2. About US$ 6.3 million cancelled from Part B (component 2) represents the initial allocation for the CIS from Part B of the project. This activity was measured only in terms of intermediate results in the original results framework and was captured by the existing result indicators. As part of the 2011 restructuring, the size and technical solution of this activity was modified and loan proceeds were reallocated to this activity in an amount of about US$ 15 million. Two new result indicators and six intermediate indicators were added to accurately reflect the change and the impact in the project. The development objective did not change as the availability of the e-government service for civil status was already assumed as part of the Project Development Objective and the scaling up of the system was envisaged by the PAD. The later cancellation of this activity of this activity in 2013 involved the reduction of the loan amount and the dropping of the result and intermediate indicators introduced in 2011. The CIS e-government service planned under the project would have increased the availability of e-government services, but its uptake at the level of population and derived benefits would not be observable in the project life time-span. Therefore, the cancellation of the CIS e-government service activity under Part B does not impact the current PDO rating which relies on actual project achievements. 32 (d) Estimated Project Costs and Actual Cumulative Cost (as of March 31, 2013) Actual % of total cumulative actual Project Component, Sub-component and Activity Est. Costs as of cumuatlive 3/31/13 costs as of 3/31/13 Component 1. Access to ICT in knowledge disadvantaged communities and improved digital literacy 1.A Improving Access by LCeNs 1.A.1. LCeN Infrastructure - Design and Supervision of Installation 1,050,000 703,451 1.A.2. LCeN Infrastructure - Equipment, Connectivity & Communications 24,100,000 24,691,843 1.A.3 Technical assistance for LCeN staff 2,600,000 4,711,932 1.A.4 Up-grade/Renewal of LCeN Equipment 7,503,350 0 1.A.5 Community Support Center (Technical and Professional Support) 550,000 528,520 1.A.6 NSP - Operating Costs 0 0 1.A.7.1 National and Local Dissemination for improving access by LCeN 75,422 75,422 1.A.7.2 National and Local Dissemination for improving access by LCeN, human resources development and e-government services 620,000 555,100 1.A.8.1 Monitoring and Evaluation for improving access by LCeN 265,000 173,745 1.A.8.2 Monitoring and Evaluation for improving access by LCeN 9,727 9,741 1.A.9. Furniture for LCENs 537,072 537,072 Total 1.A 37,310,571 31,986,825 72 1.B. Digital Literacy for Communities 1.B.1 Training for LCeNs Staff 638,120 638,120 1.B.2 ICT basic literacy training and materials development for communities 1,200,000 1,167,725 1.B.3 Training for Librarians 95,000 0 Total 1.B 1,933,120 1,805,845 4 1.C Digital Literacy for Schools 1.C.1 Teacher training 842,500 774,596 1.C.2 Education Policy Development 235,200 235,199 1.C.3 School Networking (exchanges, workshops, seminars) 85,028 85,028 1.C.4 Digital Content (Educational Software) 1,750,000 0 1.C.5.1 Technical assistance for school networking 190,000 139,534 1.C.5.2 Technical assistance for school networking 618,400 0 Total 1.C 3,721,128 1,234,357 3 Total 1 42,964,819 35,027,027 78 Component 2. Development and promotion of e-government services 2.A Portal for online Notification and Authorization for Local Services 2.A.1 Portal Application Development/IT&C Equipment/Training 700,000 679,085 Total 2.A 700,000 679,085 2 2.B Integrated Network for Issuance and Renewal of Civil Information 2.B.1 System Analysis for Central Integration Application 160,027 130,466 33 2.B.2 Development of Central Integrated Application/IT&C Equipment/Training 6,800,000 0 Total 2.B 6,960,027 130,466 0 Total 2 7,660,027 809,550 2 Component 3. Promotion of e-commerce and innovation support for MSMEs 3.A e-Community and business networking Portal 3.A.1 e-Community and business networking Portal 710,000 1,159,109 Total 3.A 710,000 1,159,109 3 3.B Matching Grant Facility 3.B.1 Matching Grant Facility 9,000,000 1,392,191 3.B.2 Monitoring of Matching Grant Implementation 200,000 205,043 3.B.3 Evaluation of Grant Proposals 40,000 39,654 Total 3.B 9,240,000 1,636,888 4 3.C National and Local Dissemination for Grant Facility 3.C.1 National and Local Dissemination for Grant facility 200,000 115,133 Total 3.C 200,000 115,133 0 Total 3 10,150,000 2,911,130 7 Component 4. Project management 4.A Project Management Unit 4.A.1. Staff Costs 4,000,000 2,234,190 4.A.2. Office Equipment 100,000 108,692 4.A.3 Office Furniture 40,000 12,187 4.A.4 Office for PMU (rent) 400,000 211,470 4.A.5 Vehicles for PMU 100,000 45,431 4.A.6 Training for PMU 320,000 62,459 4.A.7 Audit 105,000 88,741 4.A.8 Other recurrent costs 320,000 254,902 4.A.9 Technical assistance for FMS 22,829 4.A.10 Consultancy for PMU 21,000 53,455 4.A.11 Authorised translations and associated costs 5,000 632 Total 4.A 5,411,000 3,094,989 7 4.B Project Monitoring and Evaluation 4.B.1 Project Monitoring and Evaluation 1,095,000 986,320 4.B.2 Supervision of Project Monitoring and Evaluation 70,000 43,199 Total 4.B 1,165,000 1,029,519 2 Total 4 6,576,000 4,124,508 9 Advance refinance 1,838,146 1,838,146 Total Project Expenditure 69,188,992 44,710,362 0 Bank charges 1,217 Interest paid out of the Loan Account 14,096 Total Project Expenditure 69,188,992 44,725,675 34 Annex 2. Outputs by Component Component 1. Access to ICT in Knowledge Disadvantaged Communities and Improved Digital Literacy Rating: Highly satisfactory (Actual costs US$ 38 million) The objective of this component to provide access to ICT through the provision of physical equipment and connectivity and to achieve increased benefits for the communities by leveraging the access to information provided by the Local Community Electronic Networks (LCeNs) through targeted technical assistance was fully achieved, as shown by the results presented below, following the project impact chain (access-use-benefits): The following results contributed to achieving the Improved Access to ICT outcome through the provision of technical assistance, goods and training:  The objective of targeting the most K-disadvantaged communities through a competitive and transparent selection procedure, built on community ownership commitments, was achieved. Thus, the selection of communities was done in two competitive phases: (a) Pilot-phase, by which 9 communities were selected through a competitive selection encompassing 80 communities; and (b) Scale-up phase, by which an additional 246 communities were selected, based on the lessons learnt from the pilot stage. As part of the local commitment 510 staff ( 255 LCeN managers and 255 IT administrators) were hired by the local communities and financed from local budgets as opposed to loan, as initially envisaged by the project.  The objective to improve the access to ICT of approximately 45% 30 most disadvantaged communities in Romania, by establishing at least 20031 LCeNs was achieved. Thus, the project established 255 LCeNs in 208 communes and 47 small cities. Together they represent about 44% of the most disadvantaged communities in Romania and cover a population of about 1.09 million people.  The objective of implementing a LCeN multi-purpose node business model was achieved in all selected communities. Thus, in each of the 255 communities the local electronic networks that were establish link the following institutions: (i) the basic and lower secondary schools; (ii) the local administration offices (Town Halls); (iii) the public libraries; and (iv)the Public Information Access Points (PAPIs). The model was adjusted to also respond to the specific needs of the 246 villages and neighborhoods belonging to the selected communities, by the addition of a secondary node in the lower secondary schools located in those areas. In this respect the project delivered the following outputs:  255 LCeNs with nodes in: (i) 502 schools; (ii) 255 Town halls; (iii) 255 public libraries and (iv) 255 PAPIs were connected to broadband Internet (more than 100 km of fiber optic); 30 Target from PAD (page 7 and 59): “approximately 45% of most disadvantaged communities�. 31 Target from PAD and LA : “at least 200 LCeNs� 35  5,643 computer stations (4,905 desktops and 738 servers) were delivered and installed;  6,627 units of computer desks delivered and assembled;  9.579 units of chairs delivered;  492 units of storage cabinet delivered and assembled;  Operational acceptance issued for all 255 LCeNs;  510 LCeNs staff trained (2 staff/LCeN) during the LCeN installation;  Call Center established for technical troubleshooting and online assistance;  Online Monitoring for the use of LCeNs established. The following outputs contributed to Simple Use of ICT outcome through the provision of training and technical assistance to achieve:  Basic digital and other competencies:  Overall 75% of the people trained in the community use new competencies at work32;  510 persons (255 LCeN managers and 255 IT administrators) were trained in ICT fundaments – basic level in computer use (45 hours);  510 persons (255 LCeN managers and 255 IT administrators) were trained in LCeNs management, marketing and public relations (45 hours);  255 persons (IT administrators) were trained in network administration (45 hours);  255 persons (LCeN managers) were trained in Local Economic Development (45 hours).  TOSS (ICT training online support system) delivered;  200 persons (business, public administration, teachers) trained for basic digital competencies;  Over 6400 people participants in 140 info-seminars for schools. The following project outputs contributed to meaningful use of ICT through the provision of goods and technical assistance by achieving:  Sustainable LCeNs:  255 communities developed 3-year Sustainability Plans, of which 229 (or 90%) were approved by the Local Councils and are implemented;  255 people trained to prepare projects eligible for EU financing; 200 projects were prepared to increase the sustainability of the LCeNs and for local development and 250 million EUR were attracted in the communities;  103 LCeNs administrators were trained in WebDevelopmet and ICT tools applied in business; As result:  150 public-private partnerships were established in 30% of the communities;  50 reports for ICT to businesses were prepared and disseminated in communities;  Increased number of value added services provided by LCeNs (mainly through PAPIs) to different stakeholders in the communities, besides the access to 32 Project M&E - Kantor survey 36 internet, computers, fax, printing, editing copying, scanning services. In average each PAPI delivered about 15 value added services, as follows: - 248 PAPIs offered various ICT services; - 221 PAPIs provided writing, distribution, publications, newsletter for the communities; - 180 PAPIs provided counseling services for accessing financing and implementing projects with external financing; - 168 PAPIs provided space/equipment to other local organizations for various activities; - 166 PAPIs provided maintenance for network and local users, including technical support, troubleshooting hardware and software; - 156 PAPIs assisted local entrepreneurs to develop websites for e-commerce; - 148 PAPIs provided assistance for job search; - 151 PAPIs provided assistance to farmers for online submission of files for agriculture subsidies; - 157 PAPIs provided certified ICT training; - 125 PAPIs provided assistance for web design services and website administration; - 92 PAPIs provided assistance for organizing public auctions and other public events; - 90 PAPIs provided tourism information and promotion services. - 75 PAPIs provide e-payment of invoices or other support services for e- banking.  ICT Integration in Education:  Policy changes at national level: - Policy for ICT in education and Action Plan adopted and implemented by MERYS (included revision of the ICT Curricula, ICT standards for teachers, students and Inspectors, integration of ICT in the organization and administration of schools, M&E, etc) ; - Digital competency introduced in the new Education Law as one of the eight core competencies, in line with the European Frameworks;  502 primary and secondary schools were linked initially to Internet and were endowed with ICT equipment (2700 computers, printers, audio-video facilities). Due to subsequent reorganizations and consolidations of schools only 476 remained in the project; As result:  93% level of ICT integration was achieved in the project schools, as measured33 by a composite index of ten breakdown sub-indicators, as follows (further details are provided in the Results Framework and Datasheet): - 94% of schools have at least Level 1 endowment (at least one classroom endowed for the use of ICT in education) against 90% target; - 98% have audio-video facilities for ICT use in education at least at Level 1 ( audio-visual) against 90% target; - 95% of schools have access to internet ( and functional) against 90% target; - 32% of teachers are at Level 3 (ECDL Advanced level) and above in terms of certified digital competencies, against a target of 30%; - 82% of schools have ICT specialized teachers with ICT diploma, against a target of 90%; 33 Monitoring and Evaluation Self-assessment 37 - 49% of students are at least at Level 2 (ECDL Advanced level) of certified digital competence, against a 50% target; - 80% of schools have at least three classes per week that use ICT in the teaching/learning process, against a 90% target; - 25% of schools are at level 3 of integrating the Teaching Development Plan at institutional level against a target of 90% (a higher performance will be observable in the next school year as result of the TA for schools); improvement is 90% of the baseline situation; - 88% of schools with ICT to communicate with parent (MERYS, Inspectorates) institutions, against 90% target; - 100% of national outcomes for ICT in education in place (see policy changes at national level above).  Increased Teachers digital competencies and use of ICT as a tool in education, schools networking and partnerships as result of the training and technical assistance to schools: - 75% of teachers training use new skills in the teaching process and developed and used open education resources; - 687 educational projects developed for EU financing; - 5,000 DVDs were published containing with units of lessons on different educational subjects prepared by teachers; - Guide for Teachers for ICT in Education published and distributed to schools (527 pages, 15,000 copies); - Guide for Schools “ Innovation and best practices in education based on ICT integration� published and distributed to schools (96 pages, 15,000 copies); - Guide for Schools “ICT integration in schools�, published and disseminated (102 pages, 15,000 copies); - Manual “Dissemination of good practices in educational activities in schools and public libraries� published and disseminated (96 pages, 2,000 copies); - Manual “ICT utilization in education� published and disseminated (500 pages, 1000 copies); - Guide for the “Preparation and implementation of twinning programs�( Blog type project for primary schools and Tutorial about Wiki); - 14 schools concluded twinning arrangements with foreign schools; - Website was developed for school networking activities and educational resource sharing (http://scoli.ecomunitate.ro).  Increased use of ICT in community development: - Over 5,500 stakeholders participated in eCulture programs (100 entrepreneurs, 1,209 unemployed, 428 teachers, 2,000 pupils, 654 young students, 332 retirees, 264 public servants, 267 housewives, 137 librarians); - 587 digital resources were collected for the Europeana portal; - 1,281 items of local cultural heritage digitized; - 30 virtual books created by students were published on the website; - 64 databases with local cultural resources created; - 14 e-culture projects implemented. Activities and types of Outputs are listed below: Total Currency USD 38 for of the Equivalent** Project contract Life Units Total Type of of costs output Output '000 1. Access to ICT in Knowledge Disadv. Comm. and Improved Dig. Literacy 1.A. Improving Access through Development of LCeNs Design & Supervision of LCeNs Installation / 1.A.1 Consortium of online Datensysteme GmbH & reports 5 510.05 EUR 638 Cocosoft Systems Gmbh Equipment and software for LCENs lots 1, 6, 1.A.2 and 9 / JV Romsys SA & AccessNet month 35 142 USD 142 International SA Equipment and software for LCENs lots 2, 3, 1.A.2 month 35 125 USD 125 4, 5, 7, 8 and 10 / Orange Romania SA Equipment and software for 214 LCENs / JV 1.A.2 Romtelecom SA & Sysware System LCENs 214 19,851 USD 19,851 Integration SRL Equipment and software for 214 LCENs / JV 1.A.2 Romtelecom SA & Sysware System month 62 1,993 USD 1,993 Integration SRL Equipment and software for 32 LCENs / JV 1.A.2 S&T Romania SRL & AccessNet International LCENs 32 1,975 USD 1,975 SRL Equipment and software for 32 LCENs / JV 1.A.2 S&T Romania SRL & AccessNet International month 62 1,268 USD 1,268 SRL Technical Assistance for design & 1.A.3 implementation of development programs to reports 32 4,730 USD 4,730 sustainable communities Community Support Center (Technical and 1.A.5 Professional Support - non-consulting reports 38 705 USD 705 services) National and local dissemination for 1.A.7 improving access by LCeNs / Mercury reports 17 219 USD 219 Promotions SRL Developing, delivering and implementing a 1.A.7 communication and promotion strategy/ DC reports 17 612 USD 612 Communication Individual Consultant for selection process of 1.A.8 knowledge desadvantaged communities / reports 3 10 USD 10 MANUELA STANCULESCU Monitoring and Evaluation for improving access by LCeNs / Consortium of Education 1.A.8 reports 14 for Change & eRomania Gateway Association & IMAS 261 USD 261 Monitoring and Evaluation for improving access by LCeNs / Consortium of Education 1.A.8 software 1 for Change & eRomania Gateway Association & IMAS 1.A.9 Furniture for LCENs (1usd = 2,3 ron) computer desks 6,760 1.A.9 Furniture for LCENs (1usd = 2,3 ron) guest chair 9,772 1,263 RON 549 1.A.9 Furniture for LCENs (1usd = 2,3 ron) storage cabinet 502 1.B. Digital Literacy for Communities 1.B.1 Training for LCeNs Staff (1usd=2,5652 ron) reports 6 1,549 RON 604 ICT basic literacy training and materials 1B.2 reports 7 1,141 USD 1,141 development for communities 1.C. Digital Literacy for Schools 39 Technical assistance for Use of ICT in local schools and libraries / Consortium AAM 1.C.1. reports 21 2,970 USD 2,970 Management Information Consulting & Siveco Romania & TeamNet International Education Policy Development concerning IT&C integration in the pre-university 1.C.2. reports 10 235 USD 235 education system / Helsinki Consulting Group Oy Ltd. 1.C.3 School Networking (exchanges, 1.C.3 reports 3 138 RON 49 workshops, seminars) / XTEL SRL 1.C.3 School Networking (exchanges, 1.C.3 reports 5 14 USD 14 workshops, seminars) / PF Onea Emil 1.C.3 School Networking (exchanges, 1.C.3 reports 3 15 USD 15 workshops, seminars) / PF Rinderu Paul 1.C.5.1 Technical assistance for school networking reports 4 438 RON 187 The major outputs were delivered efficiently. The last activity under this component, not launched given the budget limitations (and planned to be financed from savings), the Extended Technical Assistance for Communities (est. US$4.5 million), aimed at consolidating the good results from the previous project interventions and disseminate the good practices in other communities. Its cancellation did not affect the targets negatively (see Sections 1.6-1.7). As shown below, all the targets were met with a minor shortcoming in meeting the Level of ICT integration in schools (Result #3). Additional results for the improved use of ICT in schools are expected from the ongoing Technical Assistance in schools. Unit of Indicator Name Baseline Current End Target Measure Indicator Two: Percentage Number Value 0.00 30.00 20.00 of LCeNs that operate as Sub Type Date 31-Dec-2005 28-Dec-2012 28-Feb-2013 sustainable PPP arrangements Comments 51% of LCeNs have projects approved. Moreover 80% of the mayors elected in 2012 state that will cover the Internet and other operational costs for LCeNs. Unit of Indicator Name Baseline Current End Target Measure C1 Intermediate Result 1: Number Value 9.00 229.00 At least 200.00 Number of K- Sub Type Date 31-Dec-2005 07-Feb-2013 28-Feb-2013 disadvataged communities with LCeNs Comments 229 communities are operating satisfactorily implementing the towards a sustainable sustainability business future in line with their plans. Moreover, 91% business plan of the mayors elected in 2012 appreciate that PAPI will maintain its current activities (65%) or will further develop and diversify its activities (26%). Indicator Name Unit of Baseline Current End Target 40 Measure C1 Intermediate Result 2: Percentage Value 0.00 90.00 90.00 Percentage of Sub Type Date 31-Dec-2005 07-Feb-2013 28-Feb-2013 communities that have prepared satisfactory 3- Comments 229 communities of 255 year LCeN sustainability plan Unit of Indicator Name Baseline Current End Target Measure C1 Indicator Result 3: Percentage Value 35.00 93.00 100.00 Level of ICT integration Sub Type Date 30-Nov-2007 07-Feb-2013 28-Feb-2013 in schools Comments The lower than target New indicator performance is approved as part of attributable to August 2011 unobservable impact of Project the TA in schools (sub- Restructuring indicator 8). The respective sub-indicator is expected to be fulfilled and can be reported only at the beginning of the 2013- 2014 school year Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 1. Percentage Value 71.00 94.00 90.00 Proportion of schools that Sub Type Date 30-Nov-2007 07-Feb-2013 28-Feb-2013 have classrooms endowed for the use of ICT in Breakdown Comments L0: 29%; L1: LO: 6%; L1: 55%; L2: 90% of schools at education 55%; L2: 9%; L3: 20%; L3: 17%; L4: 1%; least at Level 1 7%; L4: 0%; L5: L5:1% New sub-indicator 0% approved as part of August 2011 Project Restructuring Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 2. Percentage Value 68.00 98.00 90.00 Percentage of schools that Sub Type Date 30-Nov-2007 07-Feb-2013 30-Nov-2010 have audio-video facilities for ICT use in education34 Breakdown Comments L0: 32%; L1: L0: 2%; L1: 18%; L2: 31%; L2: 32%; 70%; L3: 11% L3: 5% Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 3. Percentage Value 54.00 95.00 90.00 Percentage of schools Sub Type Date 30-Nov-2007 07-Feb-2013 28-Feb-2013 34 New sub indicator approved as part of August 2011 Project Restructuring 41 with access to internet35 Breakdown Comments Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 4. Percentage Value 11.00 32.00 30.00 Enhanced ICT Sub Type Date 30-Nov-2007 07-Feb-2013 30-Nov-2013 competencies of Teachers36 Breakdown Comments Total: 1,511 Total: 2,031 teachers of 30% at least at teachers, of which which at: L1:776 Level 3 or above37 at: L1: 902 (38%); L2:613 (30%); (60%); L2: 450 L3:284 (14%); L4: 157 (30%); L3: 92 (8%); L5: 201 (10%). (6 %); L4: 42 Correction of (3%); L5: 25 11/30/2011 data: 17% (2%). 1,119 teachers, of which at: L1: 571 (51%); L2: 353 (32%); L3: 284 (8%); L4: 21 (2%); L5: 81(7%). Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 5. Percentage Value 56.00 82.00 90.00 Percentage of schools Sub Type Date 30-Nov-2007 07-Feb-2013 31-Dec-0131 that have ICT specialized teachers with ICT (or Breakdown Comments L0: 44%; L1: L0: 18%; L1: 43%; L2: 90% of schools at equivalent) diploma 35%; L2: 14%; 25%; L3: 10%; L4: 2%; least at level 139 L3: 4%; L4: 1%; L5: 2% 38 L5: 1% Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 6. Percentage Value 0.00 49.0040 50.00 Enhanced ICT Sub Type Date 30-Nov-2007 07-Feb-2013 30-Nov-2013 competencies of students Breakdown Comments 3,411 students 192 students have Percentage at level have certified ICT certified ICT 2 or above41 competence: L1: competence: L1: 1,123 2,649 L1 (100%). (51%); L2: 502 (23%); L3: 567 (26%) Note: error in performance reported 35 New sub indicator approved as part of August 2011 Project Restructuring. 36 Measured as percentage of teachers of total with increased level of certified digital competence. Five levels of ICT certified competencies have been defined: L1 - Foundation Level ICT Literacy Certificate (ECDL Certificate 1st level or equivalent);L2 -Intermediate Level ICT Literacy Certificate (ECDL Certificate 2nd level or at least one out of two sub-levels -or equivalent); L3 - Intermediate Level ICT Literacy Certificate (ECDL Certificate 2nd level both modules - or equivalent); L4 - Advanced Level ICT Literacy Certificate (ECDL Certificate 3rd level one or more modules - or equivalent); L5 - Advanced Level ICT Literacy Certificate (ECDL Certificate 3rd level all modules - or equivalent). 37 New sub-indicator approved as part of August 2011 Project Restructuring. 38 Indicator highly sensible to staff fluctuation in rural schools. 39 New sub-indicator approved as part of August 2011 Project Restructuring. 40 It measures the percentage of students with increased the level of ICT competence (L2 and L3). Three levels have been defined: L1 -Introduction toICTs (ECDL Certificate 1st level or equivalent); L2 - Intermediate Level ICT Literacy Certificate Foundation Level ICT Literacy (ECDL Certificate 2nd level-at least one out of two sub-levels, or equivalent); L3 - Intermediate Level ICT Literacy Certificate Foundation Level ICT Literacy (ECDL Certificate 2nd level or both modules, or equivalent). 41 New sub indicator approved as part of August 2011 Project Restructuring. 42 previously (45% instead of 320% as reported). Unit of Indicator Name Baseline Current End Target Measure ICT in education: 7. Percentage Value 32.00 80.00 90.00 Students access to ICT Sub Type Date 30-Nov-2007 07-Feb-2013 30-Nov-2013 resources Breakdown Comments L0: 38%; L1: L0: 9 %; L1: 11%; L2: L2 or more 30%; L2: 13%; 58%; L3: 16%; L4: 5%; New sub indicator L3: 19%; L4: 0%; L5: 2% approved as part of L5: 0% Correction of the August 2011 previous reporting: Project 11/30/2011: 69% of Restructuring which: L0: 16%; L1: 15%; L2:50%; L3: 12%; L4: 6%; L5: 1% Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 8. ICT Percentage Value 19.00 25.00 90.00 programming at school Sub Type Date 30-Nov-2007 07-Feb-2013 30-Nov-2013 level Breakdown Comments L0: 38%; L1: L0: 10%; L1: 37%; L2: 90% at least at 30%; L2: 13%; 29%; L3: 25%42 level 3 L3: 19% New sub-indicator approved as part of August 2011 Project Restructuring Unit of Indicator Name Baseline Current End Target Measure ICT in Education: Percentage Value 35.00 88.00 90.00 9.Communication with Sub Type Date 30-Nov-2007 07-Feb-2013 30-Nov-2013 parent institutions using ICT means Breakdown Comments L0: 26%L1: L0: 3% ; L1: 7% 90 % at least at 31%L2: 9%L3: L2: 2% ; L3: 1% Level 343 7%L4: 15%L5: L4: 20% ; L5: 67% 13% Unit of Indicator Name Baseline Current End Target Measure ICT in Education: 10. Percentage Value 0.00 100.0044 100.00 42 The impact of the TA in schools is expected to be fulfilled and can be reported at the beginning of the 2013-2014 school year. Currently in all targeted communities the main schools have Institutional Development Plans prepared with implementation planned to start as of September 2013. In addition, there is a 90% improvement of the baseline situation. 43 New sub-indicator approved as part of August 2011 Project Restructuring. 44 New sub-indicator approved as part of August 2011 Project Restructuring 43 Outcomes at national Sub Type Date 30-Nov-2007 15-Jan-2013 30-Nov-2013 level Breakdown Comments The following have been adopted: (i) ICT education policy; (ii) Action Plan for ICT in education; (iii) ICT competence required by National Curriculum; (iv) ICT Competency Standards for Teachers; (v) M&E for ICT in education. Unit of Indicator Name Baseline Current End Target Measure C1 Intermediate Result 4: Percentage Value 58.00 75.00 75.00 Percentage of targeted Sub Type Date 31-Dec-2005 07-Feb-2013 28-Feb-2013 teachers that are using the new skills in Comments classroom Unit of Indicator Name Baseline Current End Target Measure C1 Intermediate Result 5: Percentage Value 0.00 85.00 75.00 Percentage of people Sub Type Date 31-Dec-2005 07-Feb-2013 28-Feb-2013 trained in the LCeNs aquired basic digital Comments ICT training events literacy skills and are were organized during using the new skills in the 2012 for different work categories of beneficiaries (public admin. (105), farmers (156), entrepreneurs (104), children (222), other category (87) Component 2. Development and promotion of e-government services Rating (after last restructuring of Feb 2013): Moderately Unsatisfactory (Actual costs US$ 0.8 million) Based on the approved restructurings, the project delivered the agreed outputs under Component 2 to develop and promote e-government services and met the revised targets (Sections 1.7 and 2.2). 44 Total Currency Type of Units of USD 2. Development and Promotion of Government e-Services costs of the output Output Equiv.** '000 contract 2.A. Online System for Online Notifications and Authorizations of Local Businesses Portal Application Development/IT&C 2.A.1 system 1 683 USD 683 Equipment/Training 2.B. Integrated Network Civil Information and Documents Design & Implementation plan of an integrated 2.B.1 system for issuance and renewal of civil information reports 5 159 USD 159 / NESTOR LAB scarl Firstly, under Subcomponent 2A the project originally has as output that was delivered, namely the Online system for the registration of individual entrepreneurs and family associations. However, the utilization of the online system developed under the project (0.8 million) was stopped in 2011 following the adoption of a new legislative framework governing the registration of entrepreneurs and family associations that excluded the role of local authorities as facilitators in the registration process. Hence, the registration process supported by the system developed, as well as the indicators that measured improved efficiency of local authorities as facilitators were dropped as irrelevant for the newly approved registration process. Furthermore, the Online registration of entrepreneurs and family associations, targeted by the project, was implemented nation-wide in 2012 as a result of the knowledge achieved under the project through the addition of a special software module to register these categories of businesses targeted by the project. In recognition to the actual output by end project the output indicator was changed (as part of the Restructuring) to actually reflect the current situation. Secondly, based on the restructured design, the project delivered two outputs under 2B, namely (i) the business processes analysis and identification of technical and functional alternatives for implementation (as result of the TA) ; and (ii) the technical specifications for the nation-wide application of the system. The implementation of the originally designed output, namely the implementation of the CIS e-government system was much delayed initially due to concerns of security of information, and later due changes in technical solutions, changes in the sequence of implementation, and of institutions involved and foremost by the insufficient budget allocation to allow the implementation. The project was restructured three times to respond to changed context and needs: first, the June 2010 restructuring provided a 18 months extension to allow sufficient time for implementation; second, the August 2011 restructuring allowed for a nation-wide implementation of the system vs. the gradual scale up as envisaged originally by the PAD, and the utilization of the state intranet vs. the internet through reallocation of project funds and changes of the result indicators; and third, by February 2013 when the implementation of the system was dropped, the respective project funds were cancelled and results framework amended. Although the project could not implement the CIS, it built significant knowledge and strengthened the inter-institutional cooperation. These achievements allow for a now smoother implementation with EU funds, which otherwise would not have been possible. The component had been rated previously unsatisfactory because of the progress in implementing e-Government services had been practically stalled since September 2011. The procurement process for the Integrated System for the Issuance of Civil Information (CIS), estimated to about US15 million did not start, despite the efforts made to timely restructure the project and to finalize the bidding documents (the design was restructured in 2011). 45 Results Unit of Indicator Name Core Baseline Current End Target Measure C2 Intermediate Result 1. Online Text Value no yes45 yes registration of business available as Sub Date 31-Dec- 07-Feb-2013 28-Feb-2013 result of the knowledge achieved Type 2005 under the project Comments Unit of Indicator Name Core Baseline Current End Target Measure C2 Readiness of bidding documents to Text Value no yes46 yes implement CIS with EU funding Sub Date 31-Dec- 07-Feb-2013 28-Feb-2013 Type 2005 Comments . Component 3. Promotion of e-commerce and innovation support for MSMEs Rating: Satisfactory (Actual costs US$ 3.5 million) The project enhanced access of knowledge disadvantaged communities to business opportunities and access to information and accelerated adoption by firms of innovation e-business solutions. 3. Promotion of e-Commerce and Innovation Support for SMEs 3.A. Portal for Promotion of E-Commerce and Business Networking 3.A.1 e-Community portal / SIVECO ROMANIA SA portal 1 580 1,717 USD 3.A.1 e-Community portal / SIVECO ROMANIA SA month 43 1,137 3.B. Grant Facility 3.B.1 SC MATEROM SRL reports 3 223 RON 77.23 3.B.1 SC AKTIV PLUS SRL reports 2 33 RON 11.53 3.B.1 SC ADA COMPUTERS SRL reports 3 141 RON 48.97 3.B.1 SC FLAMIS SRL reports 3 66 RON 22.95 3.B.1 SC START TOPO SRL reports 2 31 RON 10.76 3.B.1 SC CONSTRAIN SRL reports 3 33 RON 11.38 3.B.1 SC GEMINI CAD SYSTEMS SRL reports 3 223 RON 77.08 3.B.1 SC ASCO TOOLS SA reports 3 192 RON 66.39 3.B.1 SC BMB MET CONSTRUCT SRL reports 3 223 RON 77.23 3.B.1 SC ARCA PRESS SRL reports 3 121 RON 41.75 3.B.1 SC BRANDSON MARKETING SRL reports 3 223 RON 77.19 45 The utilization of the on-line registration system developed with project funds was stopped in 2011 following the adoption of a new legislative framework governing the registration of local business that excluded local authorities from the registration process. The National Trade Registry provides the possibility for the local business to register on- line. 46 Due to the budgetary constraints the implementation of the CIS with project funds was canceled, in spite of the readiness of the bidding documents. 46 3.B.1 SC GAIA CONSULTING SRL reports 3 33 RON 11.28 SC METRO MEDIA TRANSILVANIA STUDII 3.B.1 reports 3 193 RON 66.76 SOCIALE 3.B.1 SC PROMOTION IDEAS CENTER reports 3 219 RON 75.85 3.B.1 SC VMB PARTENERS reports 3 218 RON 75.64 3.B.1 SC SAN SOFTWARE SRL reports 3 120 RON 41.40 3.B.1 SC SDR PROJECT SRL reports 3 33 RON 11.58 3.B.1 SC TIC MARCON SRL reports 3 33 RON 11.58 3.B.1 SC WEST CAR TRADE SRL reports 2 139 RON 48.06 3.B.1 SC ROMACTIV BUSINESS CONSULTING SRL reports 3 220 RON 76.23 3.B.1 SC AGRESS BUSINESS COMPANY SRL reports 3 152 RON 52.60 3.B.1 SC TELEVOX SA reports 3 168 RON 58.24 3.B.1 SC CALEXIS SRL reports 3 33 RON 11.58 3.B.1 SC SOFTROM GRUP SRL reports 3 33 RON 11.48 3.B.1 SC EUROKYT SRL reports 2 221 RON 76.47 3.B.1 SC EVOL CONSULT SRL reports 2 33 RON 11.59 3.B.1 SC MEDIA PLUS & ADVERTISING SRL reports 3 33 RON 11.58 3.B.1 SC PRO & EL SRL reports 3 33 RON 11.57 3.B.1 SC 2 CONNECT ROMANIA SRL reports 3 139 RON 48.24 3.B.1 SC BOTOND DOKI SRL reports 3 33 RON 11.59 3.B.1 SC UDVARHELY HIRADO SRL reports 2 124 RON 42.80 3.B.1 SC ADVENTURE CAMPS SRL reports 2 30 RON 10.39 3.B.1 SC AROBS TRANSILVANIA SOFTWARE SRL reports 3 118 RON 40.81 3.B.1 SC TRAINING & DEVELOPMENT SRL reports 3 32 RON 11.24 3.B.1 SC STO SM SRL reports 2 33 RON 16.83 3.B.1 SC XEROM SERVICE SRL reports 3 217 RON 75.14 3.B.1 SC OCOLUL SILVIC DE REGIM GHEORGHIENI SA reports 3 199 RON 69.04 3.B.1 SC BUSINESS CONCEPT SRL reports 3 31 RON 10.68 3.B.1 SC SILVAGRA SRL reports 3 195 RON 67.62 3.B.1 SC BUSINESS INFOCONS SRL reports 3 33 RON 11.59 3.B.1 SC ART DECO SRL reports 3 24 RON 8.27 3.B.1 SC MULTIPROMT SRL reports 2 24 RON 8.24 3.B.1 SC ORTODAC SRL reports 2 30 RON 10.45 3.B.1 SC EXYCOMP SA reports 3 28 RON 9.75 Monitoring of Matching Grant Implementation/ AAM 3.B.2 reports 8 205 USD 205 Management Information 3.B.3 Evaluation of Grant Proposals/ Grigore Manole report 1 13 USD 13 3.B.3 Evaluation of Grant Proposals/ Eliza Lupascu report 1 13 USD 13 3.B.3 Evaluation of Grant Proposals/ Zamfira Balan report 1 40 RON 13 3.C.1 National and Local Dissemination for Grant Facility/ Event 4 You report 6 190 USD 190 First, under Subcomponent 3A the portal was developed (http://www.ecomunitate.ro/business) and has progressively developed content relevant for the stakeholders in the communities assisted by the project. The data provided by Google Analytics show that since launching in March 2009 the portal was visited 906,520 unique visitors 1,291,178 times. The portal has sections dedicated to business, education, public administration, etc. The business section provides a virtual space for business networking, dedicated to rural entrepreneurs. 47 Second, under Subcomponent 3B 41 firms implemented e-business solutions under the Grant facility. The initial target of 300 was revised downwards to 41 projects as part of the 2011 restructuring given the availability of EU grant funding for similar objectives and funds were reallocated for subcomponent 2B CIS. The total value of the 41 projects implemented is of more than 6.8 million RON, of which the project finances 4,287,185 RON or the equivalent of US$1,095,000. All the projects were implemented on time. Third, the Subcomponent 3.2 financed the Monitoring and evaluation of the Grant Facility program. Findings and recommendations were used to simplify the EU funded manuals. Results Unit of Indicator Name Core Baseline Current End Target Measure C3 Intermediate Result 1: Number Value 0.00 1,104,995.0047 500,000.00 Number of visits in the e- Sub Type Date 31-Dec-2005 07-Feb-2013 28-Feb-2013 Portal Comments Reflects the cumulative number of visits Unit of Indicator Name Core Baseline Current End Target Measure C3 Intermediate Result 2: Number Value 0.00 41.00 41.0048 Number of grants awarded Sub Type Date 31-Dec-2005 07-Feb-2013 28-Feb-2013 Comments 47 Revised indicator as part of August 2011 project restructuring. 48 Revised target as part of August 19, 2011 project restructuring. 48 Annex 3. Economic and Financial Analysis Not applicable. 49 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Arabela Sena Aprahamian Senior Operations Officer ECSF2 Alexandra L. Drees-Gross Senior Financial Specialist FFSAB S. Gizem Eren-Baig Senior Resource Management Off CFRPA Lynn M. Gross Program Assistant EASFP Tobias Linden Lead Education Specialist SASED Ionel Lumezianu Project Officer IMTGS Mariana Doina Moarcas Senior Operations Officer ECSH2 Ana Maria Sandi Consultant ECSHD Ramin Shojai Consultant AFTRM Ireneusz M. Smolewski Senior Procurement Specialist ECSO2 Shobhana Sosale Senior Operations Officer AFTEW Alexey K. Volynets Associate Knowledge Management CESGF Supervision/ICR Arabela Sena Aprahamian Senior Operations Officer ECSPF2 Corina Alexandrescu Senior Program Assistant ECCRO Bogdan Constantin Sr Financial Management Specialist ECSO3 Constantinescu Mihaela Dumitrascu Executive Assistant ECCRO Paula Genis Operations Officer ECSPF Ionel Lumezianu Project Officer IMTGS Mariana Doina Moarcas Senior Operations Officer ECSH2 George Alexandru Information Assistant ECCRO Moldoveanu Ana Maria Sandi Consultant ECSHD Ireneusz M. Smolewski Senior Procurement Specialist ECSO2 Arben Maho Senior Procurement Specialist ECSO2 50 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY04 29.99 153.79 FY05 35.71 126.19 FY06 21.54 72.91 Total: 87.24 352.89 Supervision/ICR FY05 0.05 0.04 FY06 14.21 53.47 FY07 36.24 101.83 FY08 29.50 98.88 FY09 33.48 119.42 FY10 36.34 103.84 FY11 33.46 114.00 FY12 21.27 69.80 FY13 27.80 125.41 Total: 232.35 786.69 51 Annex 5. Beneficiary Survey Results Household and Institutional Impact Assessment Conceptual framework49 The ICT impact chain is modeled as a five-step process on two levels: individual/ household level and collective/ community level. The concept of capabilities is defined at each level on six dimensions. Individual capabilities relate to informational, psychological, social, economic, political and cultural areas, while the collective capabilities refer to informational, organisational, social and economic development, political participation and cultural identify. Each of these is linked to a set of outcome indicators that could be measured according to ICT impact. Figure 7. ICT impact chain as a 5-step process (Source: Gigler (2011). 1 2 3 4 5 Individual  Informational Access to ICTs Use of ICTs Meaningful  Enhanced  Enhanced  and  Needs  Use Informational  Human household  Assessment Capabilities Capabilities level Enhanced  Existing  ICT Simple ICT  Level and  Multiplier  Social Community  Information  Infrastructure Use purpose of  Effect Capabilities level Ecologies ICT Use Intermediation Structural barriers KEP impact at the individual and household level Realized functionings - Usage of mobile phones, computers and Internet Access to ICT has significantly improved in the communities covered by the Knowledge Economy Project (KEP), much more than in the areas were the project was not implemented. Improved access to ICT tools, such as Internet, represents a potential for enhancing people's capabilities but does not automatically lead to better and more diversified livelihoods: 'having Internet access is a necessary, but insufficient, condition for its use... Actual use of ICT, rather than access to ICT, is a critical precondition for ICT to have an impact on well-being and the knowledge on how to use the Internet is more important for poor peoples‘Internet uses, than having access to the technology within their communities (Gigler, 2009). Attached tables show usage of ICT in KEP and non-KEP areas. 49 This annex contains excerpts from the Impact Evaluation completed in April 2013. The numbering of figures and tables have been left intact. 52 Table 5. Children's (3-15 years) use of ICT Control KEP KEP KEP non- Total Urban Rural KEP Rural Total number of children 3-15 years 381 164 217 151 Number of children 3-15 years enrolled in 356 145 211 138 education % 100 100 100 100 Mobile phones Have at home 93 97 90 72 (%) Ever used 57 60 55 38 Computers (PC) Have at home 67 74 63 46 (%) Ever used 88 89 87 62 Regular use (daily or at least once a week) 76 79 73 47 Internet Have at home 51 70 39 41 (%) Ever used 73 77 71 57 School PCs (%) Access to PC at the place of education (school) 83 77 87 78 Classes assisted - regular classes (daily or at least once a week) 64 53 71 38 on PC at school - more rarely 13 17 10 17 (%) - never 23 30 18 45 Data: KEP Surveys 2012-2013. Note: Colored cells indicate significantly different values, either significantly higher or significantly lower. Regular use refers to the last 3 months before the survey. Table 6. Use of ICT of population (16-74 years) Control KEP KEP KEP non- RO EU-27 Total Urban Rural KEP Rural Number of persons 16-74 years 1,947 825 1,122 590 % 100 100 100 100 Mobile phones Heard about 99.1 99.6 98.7 100.0 Have at home 86.7 89.2 84.8 76.6 (%) Ever used 82.9 86.1 80.7 74.1 75 87 Computers (PC) Heard about 94.0 96.0 92.6 91.4 Have at home 55.3 60.4 51.6 45.9 38 67 (%) Ever used 49.2 52.8 46.5 45.6 56 79 Regular use (daily or at least once a week) 42.4 45.1 40.4 36.1 45 70 Computer use Beginner 17.0 17.1 16.9 21.4 Level of skills Medium 20.3 22.4 18.8 18.3 (%) Advanced (including expert) 10.5 12.1 9.3 5.9 Internet Have at home 48.0 57.2 41.3 37.5 36 67 (%) Ever used 45.8 47.9 44.3 41.0 52 76 Regular use (daily or at least once a week) 40.7 43.6 38.5 34.9 43 69 Internet use Beginner 14.2 12.8 15.2 17.1 Level of skills Medium 20.0 21.8 18.7 17.8 (%) Advanced (including expert) 10.4 12.0 9.2 6.1 Data: KEP Surveys 2012-2013. For Romania and EU-27, Eurostat data for 2012, accessed on March 16th, 2013; regarding the use of mobile phone, the most recent data available are for 2008 and regarding access at home, the most recent data are from 2011. Notes: Use of ICT whether at home, at work or from anywhere else and whether for private or work/business related purposes. Regular use refers to the last 3 months before the survey. The indicator of level of skills is self-declared and relates to all individuals aged 16 to 74 who have ever used a PC or Internet. Colored cells indicate significantly different values, either significantly higher or significantly lower. … 53 Enhanced informational and human capabilities - Meaningful use of Internet This section shifts from use to meaningful use defined in operational terms as regular use of Internet in the last three months for deriving real benefits in informational, social, economic, political and cultural (identity) areas. Unlike the simple use of Internet, meaningful use captures the depth, usefulness and level of expertise in Internet use. Informational area The new ICT capabilities (person‘s ability to use computer hardware, software, and ICT tools), achieved within the project, help residents of KEP communities in enhancing their informational capital through accessing Internet for information that can play a transformative role on existing human and social capabilities and thus can have an impact on their livelihood and well-being. In KEP area, the Internet use for informational purposes is as high as the national average. Searching on Internet for information about goods and services is a behavior more spread than in the control non-KEP (31% in KEP rural compared with 24% of individuals aged 16-74 years). Figure 1. Internet use for informational purposes (% individuals aged 16 to 74 years) 70 60 50 40 41 39 36 40 30 20 10 35 32 36 33 35 31 31 24 23 22 29 31 34 31 37 51 42 61 44 61 0 KEP area KEP small KEP rural Rural Small  urban Total Rural Small  urban Total urban KEP area Control  non� Romania (2012) EU�27 (2012) KEP rural Informational purposes Using the Internet for reading/downloading online newspapers / news magazines Using the Internet for finding information about goods and services Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. For Romania and EU-27, Eurostat data accessed at March 16th, 2013, where: Small urban equivalent with intermediate urbanized area (between 100 and 499 inhabitants/Km²); Rural equivalent with sparsely populated area (less than 100 inhabitants/Km²). Use of Internet within the last three months before the survey. …. Social area In KEP area, Internet use for communication purposes is as spread as in control non-KEP communities and very close to the national average. Men and women, from all ethnic groups and from all types of households, equally use Internet for communicating with family and friends, but young people use it in a considerable higher proportion than elderly.50 Ceteris paribus, significantly higher odds of using Internet for communication purposes have persons who believe that Internet represents the simplest way to keep in touch with the loved ones left abroad, with higher experience in ICT devices and tools 50 91% of Internet users aged 16-29 years compared with 73% of Internet users over 60 years old, both in KEP and non-KEP areas. 54 and more frequent use of Internet as well as individuals who travelled or worked abroad for a time or from households with migrants abroad. In this regard, easy access to Internet is not a significant determinant. The location in non-KEP rural communities (as compared to KEP communities) significantly increases the odds of using Internet for communication purposes. An analysis limited to the communities covered by KEP shows that provision of services of communication online through LCEN (at PAPI) has positively contributed (but not statistically significant) to the enhancement of the residents' communication capabilities.51 However, in interviews, the new communication opportunities created by the project were highly valued, especially in communities with high rates of temporary migration for work abroad. Figure 2. Internet use for horizontal communication purposes (% individuals aged 16 to 74 years) 70 60 50 38 40 37 35 40 30 20 10 37 23 33 25 38 24 35 24 21 17 42 20 33 25 53 32 64 36 0 Rural Rural Small  urban Total Rural Small  urban Total Rural Small  urban Control  non� KEP area Romania (2011) EU�27 (2011) KEP Horizontal communication purposes Communication Participating in social networks (creating user profile, posting messages or other contributions to facebook, twitter, etc.) Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. For Romania and EU-27, Eurostat data accessed at March 16th, 2013, where: Small urban equivalent with intermediate urbanized area (between 100 and 499 inhabitants/Km²); Rural equivalent with sparsely populated area (less than 100 inhabitants/Km²). Use of Internet within the last three months before the survey. The social dimension of well-being covers also education and health, areas in which improved informational capabilities can play a transformative role in enhancing human capabilities. With respect to education, the meaningful use of Internet relates to strengthening of individual agency in terms of raising awareness on the existing educational opportunities, enabling people to reach higher levels of education, and providing new opportunities of adult education and vocational training. In this sense, the KEP has not limited to improving ICT capabilities, but focused on enhancing informational capabilities by providing through LCEN various training courses (online or not) highly relevant for livelihood, besides digital literacy training. 'Here we are in the PAPI room at Prundeni, Vâlcea County. The KEP brought six computers and we have managed to equip the room with another 16, a Smart table, and a video projector. We held courses this year for more than 1,000 people: unemployed, civil servants, businessmen, farmers and so on, on various professional skills: IT, enterprise, English, hairdressing, European 51 Tested on the basis of the model showed in table A.15 to which we added a dummy variable regarding participation at local PAPI in services of communication online, applied only for Internet users from KEP area (N=818). This model adequately fits the data and Nagelkerke R2=0.53. Participation in the relevant PAPI service obtains: Exp(B)=1.86 at Sig.=0.125. 55 projects … The demand for places on these courses has been very, very high. At the moment we have a waiting list of 67 people for various skills.' (PAPI manager, June 2013) Figure 3. Internet use for educational purposes (% individuals aged 16 to 74 years) 40 35 30 25 18 20 16 16 15 10 10 5 10 3 15 5 17 4 16 5 17 n.a. 37 0 25 5 23 4 28 4 29 5 0 Rural Rural Small urban Total Rural Small urban Total Rural Small urban Total Control non� KEP area Romania (2011) EU�27 (2011) KEP Training and education purposes Using the Internet for looking for information about education, training or course offers Using the Internet for doing an online course Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. For Romania and EU-27, Eurostat data accessed at March 16th, 2013, where: Small urban equivalent with intermediate urbanized area (between 100 and 499 inhabitants/Km²); Rural equivalent with sparsely populated area (less than 100 inhabitants/Km²). n.a. - Data not available. Use of Internet within the last three months before the survey. Thus, 16% of individuals aged 16 to 74 years from KEP area use Internet for educational purposes compared with only 10% in non-KEP rural communities, but with 25% national average. At the level of small cities it is observed a high gap between KEP communities with only 17% of individuals using Internet for looking for information about education, training or course offers, compared with 37% at the national level.52 All other things being equal or kept constant, the odds of occurrence of Internet use for educational purposes are higher in more developed rural communities. The location in KEP communities does not influence significantly. In addition, an analysis limited to KEP communities indicates that participants to training programs (either in digital literacy or in other skills) delivered through LCENs (at PAPI) have a somewhat higher probability to use Internet for educational purposes, which however is not statistically significant.53 With regard to health, enhanced informational capabilities are linked to better access to quality health services and improved health related knowledge. In this respect, in some communes, the project played through PAPI an intermediation role between community and providers of medical services (private or public) not available within community. Figure 4. Internet use for health related purposes (% individuals aged 16 to 74 years) 40 52 35 30 official statistics at the country level registered for this indicator a leap from 13% in 2010 to 37% in 2011. We The 25 have20no explanation 15 for this evolution. 18 17 18 15 10 4 3 4 5 14 2 18 16 17 19 36 29 32 38 38 53 Tested 0 on the basis of the model showed in table A.15 to which we added a dummy variable regarding participation at local PAPI in courses Rural in IT or Rural any other skills Small urban (with or without Total Rural certification, Small urban courses Total online or not), applied Rural only for Total Small urban users   InternetControl non� KEP area (N=859). from the data This model adequately fitsRomania KEP area and Nagelkerke R2=0.46. Participation  (2011) EU�27 (2011) in PAPI services obtains: Exp(B)=1.65 at Sig.=0.091. the relevantKEP Health related purposes Using the Internet for seeking health information Using the Internet for e�health services 56 Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. For Romania and EU-27, Eurostat data accessed at March 16th, 2013, where: Small urban equivalent with intermediate urbanized area (between 100 and 499 inhabitants/Km²); Rural equivalent with sparsely populated area (less than 100 inhabitants/Km²). Use of Internet within the last three months before the survey. The proportion of individuals aged 16 to 74 years who use Internet for health related purposes is similar to the national rural average in KEP rural communities, but much lower in KEP small cities (16% compared with 36%). …. Overall, 25-26% of individuals aged 16 to 74 years from KEP communities (rural and urban) use Internet on a regular basis for educational and/or health related purposes. The corresponding share in non-KEP rural communities is only 19%. Hence, the propensity of using Internet for deriving benefits in social area is significantly higher in KEP communities compared with the other rural communities. Moreover, these Internet users make a distinctive note among the KEP residents as they assess considerably more positive the benefits of the project in the social area, in particular in rural communities. For example, 'KEP contributed to improvement of local school to a large or very large extent' believe 81% of the Internet users for educational or health related purposes from KEP rural communities compared with 45% of the other residents (users or non-users). Or, 'KEP contributed to enhancing access to medical services of better quality' consider 44% of Internet users for educational or health related purposes compared with only 22% of the other residents of KEP rural communities. In conclusion, on the social dimension, the project succeeded at a large extent to close the knowledge gap between the rural disadvantaged communities (at the project outset) and the non-disadvantaged ones by expanding the informational capabilities of residents. By contrast, in the small urban communities, the results are good with respect to enhancement of communication capabilities, but limited regarding education and health areas. The value-added services provided through LCENs (online communication, courses and trainings, health services) have contributed in a positive manner to the ICT multiplier effect on residents' communication and human capabilities, yet still insufficient for making a statistically observable change. People with higher capitals benefited more than the disadvantaged groups, but: (a) the disadvantaged groups are somewhat better than they would have been if the project would not have been implemented (in KEP compared with non-KEP communities);54 (b) the project has not explicitly targeted the disadvantaged groups. Economic area The KEP focused on capacity-building and provided a variety of employment services and locally tailored services for businesses and farmers. Nevertheless, the use of Internet for economic purposes is still very low, but it has almost reached the national averages. 54 For example, among the Internet users of Roma ethnicity (16-74 years), while in KEP communities one in every two of them use the Internet for educational or health related purposes, in non-KEP communes the proportion is virtually zero. Similar, among the Internet users from households in relative poverty, the proportion of those using the Internet for educational or health related purposes is 52% in KEP communities compared with 32% in non-KEP communes. 57 Figure 5. Internet use for economic purposes (% individuals aged 16 to 74 years) 50 39 38 40 33 29 30 20 13 11 15 18 17 16 6 7 8 7 9 10 5 2 3 4 3 3 2 5 3 3 2 1 2 1 2 1 1 2 0 Rural Rural Small urban Total Rural Total Rural Small urban Total Control non� KEP area Romania EU�27 KEP Using the Internet for looking for a job or sending a job application (2011) Using the Internet for selling goods or services (2012) Using the Internet for Internet banking (2012) Having ordered/bought goods or services for private use over the Internet (2012) Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. For Romania and EU-27, Eurostat data accessed at March 16th, 2013, where: Small urban equivalent with intermediate urbanized area (between 100 and 499 inhabitants/Km²); Rural equivalent with sparsely populated area (less than 100 inhabitants/Km²). Data for small urban area at the country level are not available. Use of Internet within the last three months before the survey. Using Internet for seeking a job or sending an application depends foremost on the employment status and, to a smaller extent, on human capital and preferences. Thus significantly higher odds of using Internet for seeking a job correspond to the unemployed but also to any employed persons (other than white collar) of active age, persons who travelled or worked abroad after 2000 (or belong to a household with migrants for work abroad), particularly Romanians,55 as well as to people with medium or advanced Internet skills, and to individuals who believe that computer knowledge is vital for finding/ keeping a job. The location in KEP area is not a significant predictor. However, within the KEP area, the participation in active labor market services delivered through LCENs (at PAPI) increases significantly the odds of using Internet for seeking a job.56 Selling goods and services over the Internet has significantly higher odds to occur among employers and non-agricultural self-employers,57 particularly Romanians (as compared to other ethnic groups),58 people with more advanced Internet skills, from households with more diversified livelihoods (remittances, incomes from agriculture) and a rather solid 55 In rural areas (KEP or non-KEP), virtually no Roma use Internet for seeking a job, even if they are regular Internet users. 56 Tested on the basis of the model showed in table A.17 to which we added a dummy variable regarding participation at local PAPI in active labor market services (information on the available jobs, job fairs, assistance in job searching and in cv editing etc.), applied only for Internet users from KEP area (N=841). This model adequately fits the data and Nagelkerke R2=0.30. Participation in the relevant PAPI services obtains: Exp(B)=4.01 at Sig.=0.000. 57 In KEP as well as in the non-KEP areas, farmers do not use Internet for selling their products even if they use Internet regularly. This is most probably the result of the fact that people who self-declare farmers focus on subsistence agriculture with no plans, strategies or willingness to sell the surplus, which is most often shared with the family members moved to cities. 58 Both in rural and urban areas, KEP or non-KEP, virtually no Roma use Internet for selling goods and services, even if they are regular Internet users. 58 economic situation (accumulated wealth).59 Ceteris paribus, individuals from KEP area have considerable lower odds to use Internet for selling goods or services as compared to people from the non-KEP area. Nonetheless, within KEP communities (rural and urban), the marketing services for local products (including traditions and tourism) have made a difference by increasing significantly the odds of Internet use for this purpose.60 'We have initiated another small project, which links local culture and traditions with business. Thus, we promote our traditional trades - weaving of rushes. On the one hand, we set at PAPI an online shop for helping our folk craftsmen, but also to Luncavita's image. On the other hand, the folk craftsmen started to deliver courses at the local school, in order to revive traditions among commune's children and young. [...] Also, in our commune we have a large community of beekeepers (self-employed), more than one hundred people. Through LCEN we helped, so now, all of them applied for smaller or larger programmes in their field.' (Mayor, Luncaviţa commune, Tulcea, June 2013) 'We make all kinds of canned fruits and vegetables. We prefer direct selling, by far, but the distance to the city made everything very complicated before the project. So, as soon as Internet became available, we set, with real success I would say, an online shop. Other thing, we are in a village, so we don't have a bank. Here, also, the Internet solved the problem, now we have Internet banking. Hence, the LCEN has been essential for our business and our lives.' (Local entrepreneur in Saschiz commune, Mureş, June 2013) Internet banking has considerably higher odds at the level of employers and non- agricultural self-employed with better skills and more frequent use of Internet, particularly Romanians (as compared to other ethnic groups), 61 for people from households with accumulated wealth as well as for the residents of better developed communes (as compared with residents of small cities or poor communes). The analysis on KEP area shows that the online banking services provided through LCENs have played a significant role in this area.62 E-commerce is also a rare behavior in the entire country and not only in KEP area. E- commerce is more frequent among better-off people, Romanians, persons who travelled/ worked abroad after 2000 (or have migrants abroad within household), employed, 59 For example, in rural areas (KEP as well as non-KEP), out of the Internet users (16-74 years) from poor households less than 3% use Internet for selling goods or services compared with 14% in non-poor households. 60 Tested on the basis of the model showed in table A.17 to which we added a dummy variable regarding participation at local PAPI in marketing services, applied only for Internet users from KEP area (N=828). This model adequately fits the data and Nagelkerke R2=0.35. Participation in the relevant PAPI services obtains: Exp(B)=9.49 at Sig.=0.003. 61 According to the KEP survey, virtually no Roma use Internet banking. The same situation is recorded for farmers and people from poor households. 62 Tested on the basis of the model showed in table A.17 to which we added a dummy variable regarding participation in online banking services provided through LCENs, applied only for Internet users from KEP area (N=828). This model adequately fits the data and Nagelkerke R2=0.46. Participation in the relevant PAPI services obtains: Exp(B)=21.34 at Sig.=0.000. 59 unemployed and students (as compared to pensioners and economically inactive people), with better and deeper knowledge in Internet, from households without children, located in rural communities. The KEP shows that the existence within community of assistance services regarding e-commerce and of support services for farmers regarding agricultural machinery and equipments at competitive prices increases significantly the odds of Internet use for e-commerce purposes.63 In total, 17-19% of individuals aged 16 to 74 years from KEP communities (rural and urban) use the Internet on a regular basis for deriving benefits in economic area. Higher likelihood to use Internet in such a manner have entrepreneurs and non-agricultural self- employed as well as people who travelled abroad (or from households of migrants for work abroad), with advanced Internet skills. The Internet use for economic purposes is still at a low level and so the multiplier effect on KEP residents' economic well-being is limited. As Gigler (2011) emphasized, due to the lack of competitive and transparent markets in many rural areas, ICT programs often fail to improve the negotiating power of small-scale farmers in local and regional markets, to significantly reduce the high transaction costs or to have any significant positive impact on their income. This appears to be also the case of the KEP: older farmers (who predominate) tend to limit to subsistence agriculture, while younger farmers seek a job in a non-agricultural sector, within the country or abroad. So, farmers from KEP communities have low likelihood to use Internet for economic purposes other than job seeking, even when have basic skills and access to technology. 63 Tested on the basis of the model showed in table A.17 to which we added a dummy variable regarding participation in trade related services provided through LCENs (assistance services regarding e-commerce and of support services for farmers regarding agricultural machinery and equipments at competitive prices), applied only for Internet users from KEP area (N=856). This model adequately fits the data and Nagelkerke R2=0.39. Participation in the relevant PAPI services obtains: Exp(B)=7.695 at Sig.=0.000. 60 Table 1. Perceptions about KEP benefits in economic area (% category) The other KEP residents Internet users for economic aged 16 to 74 years purposes KEP KEP KEP KEP KEP KEP Total Urban Rural Total Urban Rural Number of individuals 16-74 738 314 424 135 68 67 % 100 100 100 100 100 100 KEP improved Have not heard about KEP 46 57 38 21 34 7 people's chances Do not know 12 9 14 8 9 7 to find a job ... To a small or very small extent 13 8 16 24 15 34 To a large or very large extent 29 26 31 47 43 51 KEP improved Have not heard about KEP 46 57 38 21 34 7 farmers' chances Do not know 14 13 15 12 16 7 to sell their goods/ To a small or very small extent 13 10 16 24 21 28 buy better 26 19 31 43 29 57 To a large or very large extent materials Data: KEP Survey (May 2012). Notes: In each household a single person (randomly selected) responded to these questions and only 873 out of the 985 respondents are aged 16 to 74 years. Use of Internet within the last three months before the survey. Nevertheless, KEP played a significant role in enhancing meaningful use of Internet through the provision of services targeted mainly to businesses and non-agricultural self- employed. Table 7 provides additional evidence that individuals using Internet for improving their livelihood strategies assess the project benefits in economic area in much more positive terms than the other KEP residents. Political area - Vertical communication A previous section has already shown that KEP succeeded to strengthen the participants' communication capabilities, which has resulted in enhancement of horizontal communication with family members, friends and social networks from other communities. This section refers to the vertical communication between KEP residents and the local and central authorities. In this sense, ICTs frequently only play a limited role in enhancing the transparency of government institutions due to the lack of the required cultural and organizational change within government institutions themselves. While ICTs can act to some degree as a catalyst for enhanced governance, a fundamental change of behaviours and attitudes by politicians and government officials alike is necessary to enhance the accountability and transparency of government institutions. Central to the issue of good governance is the lack of and not the lack of access to ICTs' (Gigler, 2011, p. 20). Some KEP communities have made significant efforts in improving information accessibility and have developed local e-government services. However, such initiatives of good governance have depended to a large extent on the local authorities' attitude towards communication and ICT. Hence, the use of e-government services has significantly increased during the project, but it is highly unevenly distributed among communities depending on the availability of such services. 61 'The citizens, all of them farmers, locals, young and old, have been able to make applications for farming subsidies at the Public Information Access Point (PAPI). Citizens have learned what it’s about: they come and find out that a measure has been passed in agriculture, tourism, or whatever. And they come and apply. A company obtained funding of two and a half million Euros for a project, and most of the application was done via Knowledge Economy. Now they’re building silos.' (Mayor, Luncaviţa commune, Tulcea, June 2013) 'The biggest success of the PAPI centre is that it allows access to the national tax payments site. Any citizen from Luncaviţa can pay his local taxes from wherever he is in the world. So, Luncavita is ahead of many large cities from the country, even ahead of some sectors of Bucharest.' (PAPI Manager, Luncaviţa commune, Tulcea, June 2013) 'Here we have an InfoTouch for people from the village, where they can keep a track of information concerning their properties (land and house). They can find their file in our archive and access documents such as building permits, certificates, property deeds, applications…' (PAPI Manager, Prundeni commune, Vâlcea, June 2013) 'Everybody has benefitted of PAPI and that’s a very good thing, when it comes to the annual land register documents (for farming subsidies), which otherwise cost a lot and use up a lot of time to reach the city offices. Now we can do it here and PAPI people help us in using the Internet …' (Farmer, Saschiz commune, Mureş, June 2013) 'Another benefit for the whole community is effective communication. [...] We use the Internet, the official webpage of Saschiz village, for example, to post editions of the local newspaper, to post news and announcements about what is happening in the community, to inform citizens about the Local Council decisions. Everything takes place in a transparent way.' (Mayor and PAPI Manager, Saschiz commune, Mureş, June 2013) Overall, the use of Internet for interacting with authorities is four times higher in KEP area compared with the non-KEP communities, yet lower than the 2012 national average.64 Figure 6. Internet use for e-government purposes (% individuals aged 16 to 74 years) 50 (A) Using the Internet for obtaining information from public authorities web sites (last 12 months) 43 43 45 (B) Using the Internet for downloading official forms (last 12 months) 40 35 35 (C) Using the Internet for sending filled forms (last 12 months) 31 30 Use of eGovernment services (last 12 months) for (A) and/or (B) and/or (C) purposes 25 19 20 15 12 13 12 15 10 6 7 3 3 5 0 Rural Rural Small Total Rural Small Total Rural Small Total Rural Small Total urban urban urban urban Control KEP area Romania (2011) Romania (2012) EU�27 (2012) non�KEP 64 Figure 17 shows that the national average of the use of e-government services jumped from 7% in 2011 to 31% in 2012. At a meeting held by the World Bank, officials from various ministries provided a series of possible explanations such as the online services for Bacalaureat (available before 2011) or those related to the European subventions for farmers. However, there is no clear evidence or studies to grasp this substantial growth. 62 Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. For Romania and EU-27, Eurostat data accessed at March 16th, 2013, where: Small urban equivalent with intermediate urbanized area (between 100 and 499 inhabitants/Km²); Rural equivalent with sparsely populated area (less than 100 inhabitants/Km²). Use of Internet within the last twelve months before the survey. In the control non-KEP group, virtually no farmer, unskilled worker, unemployed, individual with no or low formal education, Roma or person from poor households used e-government services during the last year, even if he or she are regular users of Internet. By contrast, within KEP communities, the participation of these disadvantaged groups into communication with authorities via Internet exist, although it is lower compared with the better-off social groups.65 Ceteris paribus, higher odds to use e-government services have the better educated people, white collars, grey collars, employers and non-agricultural self-employed, particularly from rural communities (as compared to small cities). However, the most influent predictor is individual's preference for online rather than face to face interaction with public administration and public services. The location in KEP communities increases significantly the odds of using Internet for communicating with authorities either for downloading information and official forms or for sending filled forms. Thus, the project has been successful in strengthening the communication capabilities of participants both regarding the horizontal and the vertical dimensions. The project demonstrates that besides expanding the available e-governance services, the existence within community of support services for filling in official forms and especially of assistance services for entrepreneurs (information regarding available financing sources, assistance for elaboration of projects to be submitted to various donors, European in particular, and assistance for labor force recruitment) increases significantly the odds of using the e-government services via Internet.66 These results are even more valuable considering that the dominant perception in Romania is that the public institutions are inefficient and corrupt, which seriously undermines empowerment (human agency). Actually, among the individuals who use e- government services prevail the positive opinions according which the project contributed 'to a large or very large extent' to the improvement of communication between citizens and authorities, of the knowledge about local authorities' actions, as well 65 In KEP area, for example, the proportion of Internet users aged 16 to 74 years that in the last 12 months used one or more e-government services varies from 9% of Internet users with primary education at most to 48% of University graduates. 66 Tested on the basis of the model showed in table A.18 to which we added two dummy variable regarding (1) participation in support services related to e-government and (2) participation in assistance services for entrepreneurs provided through LCENs, applied only for Internet users from KEP area (N=859). This model adequately fits the data and Nagelkerke R2=0.71. Participation in the relevant PAPI services obtains: (1) Exp(B)=1.225 at Sig.= 0.695; (2) Exp(B)= 3.453 at Sig.=0.02. 63 as to the enhancement of both access and quality of public services delivered by mayoralty. Cultural (identity) area In the cultural area, KEP provided through LCENs (at PAPI) assistance services for web pages. Additionally, it offered support for all communities willing to create their own website and in the same time it created the portal www.ecomunitate which has posted material about each of the 255 covered communities. As a result, many KEP communities created their own website and have developed a series of cultural activities (see chapter 4.2.1 section on cultural identity). These activities carried out at the community level have reflected to the level of individuals. Thus, the Internet use for cultural purposes in KEP rural communities is significantly higher than in non-KEP area, but also above the national average. In small cities, the results are close to the national average. Figure 7. Internet use for cultural (identity) purposes (% individuals aged 16 to 74 years) 30 25 20 15 10 5 9 15 11 13 9 13 15 21 25 26 0 Rural Rural Small urban Total Rural Small urban Total Rural Small urban Total Control non� KEP area Romania (2012) EU�27 (2012) KEP Uploading self�created content to any website to be shared Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. For Romania and EU-27, Eurostat data accessed at March 16th, 2013, where: Small urban equivalent with intermediate urbanized area (between 100 and 499 inhabitants/Km²); Rural equivalent with sparsely populated area (less than 100 inhabitants/Km²). Use of Internet within the last three months before the survey. Higher odds of using Internet for uploading self-created content to any website to be shared correspond to younger people, particularly pupils and students, with higher education, from better-off households, individuals with medium or advanced Internet skills, who use frequently Internet, from more developed rural communities. The location in KEP area is not a significant determinant. Nevetheless, the analysis on KEP communities shows clearly that participation in the assistance services for web pages provided through LCENs (at PAPI) plays a significant role in increasing the odds of using Internet for cultural purposes.67 67 Tested on the basis of the model showed in table A.20 to which we added a dummy variable regarding participation in assistance services for web pages, provided through LCENs, applied only for Internet users from KEP area (N=859). This model adequately fits the data and Nagelkerke R2=0.42. Participation in the relevant PAPI service obtains: Exp(B)= 6.895 at Sig.=0.002. 64 Meaningful use of Internet and social structure The previous sections has examined whether the KEP succeeded in enhancing all four components 68 of informational capabilities and to what extent and under which conditions the advanced informational capabilities improved people‘s human and social capabilities as well as in which area the meaningful use of ICT has indeed played a significant role. We have seen that the services provided within the project through LCENs (at PAPI) has played a significant role in increasing odds of meaningful use of Internet in the social, economic, political and cultural areas. Taking into consideration the area of meaningful use, the population may be grouped in four significantly different types: (1) non-users of Internet; (2) users of Internet only for leisure purposes or not regular; (3) regular use of Internet only for informational and/or horizontal communication purposes (including leisure activities); (4) regular use of Internet for other meaningful purposes in order to derive benefits in social, economic, political or cultural areas (including leisure, informational and/or horizontal communication purposes). The distribution of population across these four types shows that, in KEP area, the meaningful use is more extended than in the non-KEP communities. Furthermore, while within the non-KEP communities predominate individuals using Internet only for informational and communication purposes, in KEP communities, the meaningful use tends to cover more areas of life. Figure 8. The distribution of population aged 16 to 74 years by types of Internet use 70 59 KEP Total 55 56 60 53 KEP Urban 50 KEP Rural 40 33 35 33 Control non�KEP Rural 25 30 20 11 13 10 9 10 2 2 2 2 0 Only for information and/or to For other purposes related to the communicate with relatives, social, economic, politic or friends cultural areas Have never used Internet Use Internet only for leisure Meaningful use of Internet and/or not regularly Data: KEP Survey: N=1,947 persons present in KEP communities (1,122 rural and 825 small urban) (May 2012) and for the control non-KEP group N=590. Use of Internet within the last three months before the survey. Four types of Internet use reproduce at a large extent the existing social inequalities, covering the social structure from the underprivileged to the most privileged strata. At one extreme, the disadvantaged groups of population - over 50 years old, women, people with no or low formal education, farmers and economically inactive people, poor, geographically disadvantaged households, peripheral communities, underdeveloped 68 (i) ICT capability (a person‘s ability to make efficient use of computer hardware, software, and ICT tools); (ii) information literacy (a person‘s ability to collect, process, evaluate, use and share information with others within her/his own socio-cultural context); (iii) communication capabilities and (iv) content capabilities. 65 communes - tend to concentrate among people who have never used the Internet. At the other extreme, the younger people (16-39 years), men and upper social strata have significantly higher likelihood to use the Internet in a meaningful way for deriving benefits in various areas. The better-off used their human, economic, social and symbolic capitals for enhancing their informational capabilities, which in turn have had a multiplier effect on the initial capitals, whereas the worst-off have been able to extract only lower benefits for improving their fragile well-being. Nonetheless, the comparison between KEP and non-KEP area (presented in the previous sections) indicates that the disadvantaged groups, in particular the poor, Roma and the low educated people, would have been worse if the project would not have been implemented. Meaningful use of Internet for other purposes than information and/or communication is basically an effect of human capital, Internet access and Internet skills. Ceteris paribus, higher odds of meaningful use of Internet have the unemployed, white collars, employers, non-agricultural self-employed, pupils and students, grey collars, as well as people who travelled or worked abroad after 2000, with medium or advanced Internet skills, using Internet more frequently, in particular from rural communities. In addition, the location in KEP area increases significantly the odds of meaningful use, specifically due to the services delivered through the LCENs, at PAPI.69 Better access to information and enhanced informational capabilities can enable people to make choices in their lives in various areas and so it contributes to their empowerment. Gigler (2011, p. 19) showed that the enhanced proficiency to use ICT have significant and direct positive impact on people‘s psychological well-being - individual agency and self-esteem, particularly for people who belong to the most vulnerable groups, such as women and youth. Our qualitative data fully confirm this result. Furthermore, KEP survey shows that participants to the project perceive as benefits of KEP: a sense of participation in modern world, a better public image of their community as well as an improved propensity to community participation. In this sense, among the KEP residents, the most positive assessments come from individuals who use Internet in a meaningful way for deriving benefits in various areas of life. KEP impact at the community level Enhanced social capabilities Perceived benefits of the KEP The focus of this section is to assess whether or not KEP has led to significant improvements in the economic, social or political spheres, according to the local 69 Tested on the basis of the model showed in table A.22 to which we added a variable regarding the number of services provided through LCENs in which the individual participated, applied only for Internet users from KEP area (N=868). This model adequately fits the data and Nagelkerke R2=0.42. Participation in the PAPI service obtains: Exp(B)= 1.140 at Sig.=0.003. 66 stakeholders' perceptions. In this respect, the KEP survey collected opinions from population, but also from mayors and PAPI managers, regarding the benefits for the community. About a half (53%) of the population aged 16 years or more from KEP area have heard about the project running in their community, with a large discrepancy between communes (60%) and small cities (only 44%). Most of those who have some information about the project tend to assess that the project has brought 'medium', 'large' or 'very large' benefits for their communities. Figure 9. Perceived global benefits of KEP (% persons 16 years or over) KEP rural KEP urban 16 11 No 40% 7 7 Yes No Yes 44% 56% 3 60% 1 8 3 6 16 21 7 Don't Very low Low Medium Large Very large know benefits benefits benefits benefits Assessment  of KEP  benefits  for local  people Heard that KEP  has  run in his/her community Heard that KEP  has  run in his/her community Data: KEP Survey (May 2012). N=985 persons (554 rural and 431 small urban). The difference between pies is significant. The difference between bars is not. From the population point of view, the most frequently mentioned benefits of the project refer to improved local schools, enhanced level of civilisation and openness to the world, as well as a better public image of the community, both in rural and urban communities. Individuals using Internet in a meaningful way tend to assess the KEP benefits in significantly more positive terms. According to the mayors and PAPI managers, the project has had considerable benefits related both to the population empowerment and the capacity of local institutions. Thus, more than 60% of mayors and PAPI managers declare as 'large' or 'very large' benefits of the project: improved local schools, enhanced capacity of local authorities to attract funding, improved quality of services provided by mayoralty, better public image of the community, enhanced access to information of local institutions and businesses on funding opportunities, and improved level of civilization and openness to the world of the citizens. In most communities, KEP benefits are considered (very) high by all local stakeholders. In about a quarter of communities, the institutional representatives assess KEP much positively than the general population, in agreement with the users of Internet in a meaningful manner. These communities are small cities. 67 Informational area The dominant informational pattern in Romania, both in rural and urban area, is to inform from mass media, more precisely from the television. The traditional way of obtaining information through social networks is still present, but it represents the main source of information for less than 3% of the population even in the rural areas of the country. In the last years, however, the increase in Internet proficiency has determined a change of this pattern, more and more people completing the information from television by using the Internet or even changing the main source of information from television to the Internet. This process is highly visible in the KEP area. While the non-users of Internet rely solely on mass media or social networks, more than half of the individuals using Internet for deriving benefits in various areas declare the Internet as their main source of information and/or their main source of learning new things. In parallel, a significant change in attitudes towards the Internet has been developing. People who have never used the Internet share a dominant opinion rather negative: 'Internet is not for people like me', hence 'learning to use a PC is a waste of time', even more so given that problems with public institutions can be solved only through interactions face to face and 'children who use a PC ruin their eyes and become uncontrollable'. On the contrary, most individuals using Internet in a meaningful way believe that: 'without knowing to use a computer one has no chance today' as 'via Internet is much easier to keep in touch with the loved ones working abroad' and the 'new ICT technologies can improve access to public services for people like me'. The information flows within KEP communities has improved mainly due to the PAPI nodes of LCENs as these provided guidance and support for population not only to access information, but also to understand, to interpret and to use it in a beneficial way. The communication with community members left for work abroad has been significantly eased, the interaction between citizens and local authorities has been strengthened, and the general level of information regarding governmental programs and policies has increased. This evolution is better perceived and more valued in the rural communities. Organizational area By design, KEP involved a rather high participation of communities, community leaders and institutions, since the preparation phase. The ICT infrastructure was implemented in collaboration with the local authorities and supposed a community contribution. 70 Capacity building of the local institutions was fostered during all phases of the projects, not only with equipments and financing, but with trainings of the local staff in various 70 The local communities that applied for KEP had to participate with a contribution including: (a) spaces for LCEN, (b) cover the operational and maintenance costs of LCEN as well as (c) the wages of LCEN staff (IT manager and PAPI manager). 68 areas (besides ICT) as well as continuous support and guidance during the process.71 The project had central and unified standards, but a flexible structure which allowed the participating communities to develop services adjusted to the local context and needs. Specifically for these reasons, the local stakeholders consider that KEP led to an increase of both efficiency and transparency of the local organizations (to a smaller or larger extent, from a community to another). In rural communities the assessments in this regard are considerably more positive compared with the small cities. Within KEP, local institutions expanded their capabilities to communicate and exchange knowledge with other organizations in different communities. Firstly, KEP communities formed a network which periodically changed informations among them, in particular at the PAPI level (based on the portal www.ecomunitate.ro). Secondly, the community- based organizations (formal or informal associations or NGOs) could strengthen their capabilities. For instance, in some communities, the PAPI provided assistance to informal community-based organizations to formalize so that to become eligible for funding applications. Finally, through a series of community development activities, the schools and public libraries received support in building partnerships with organizations from other communities. The monitoring reports of the project (Kantor, 2012) show that the number of joint initiatives, joint projects or partnerships, after significantly increased in the first years of the project, it has a slightly downward trend in 2011-2012. Nevertheless, in nearly all KEP communities at least a project was developed in cooperation by the local organizations (in particular, by the LCENs nodes - PAPI, Town Hall, school and public library) or in collaboration with LCENs nodes from other KEP communities, and in about three quarters of KEP communities, partnerships with different non-KEP communities (inter-school, inter-library etc.) are also functional. 'Our local network has four functioning points: the Public Information Access Point - PAPI, the Local Administration, the Library, and the School. It works like that: the library holds digital information, the school uses the library database, we utilise the capacities from the school, and the PAPI centre is nothing less than the brain of this community.' (PAPI Manager, Luncaviţa commune, Tulcea, June 2013) 'I can tell you, we are champions in attracting European funds and so there are very many mayors from all around Romania who come to visit me or call me because they’ve read or heard about the KEP, and I always have an advice for them. Many times I’ve even helped them draft their funding applications.' (Mayor, Prundeni commune, Vâlcea, June 2013) 71 At the central level, it was established a Community Support Center, call-center which provided technical assistance and support for the whole KEP network. In the same time, a team of community facilitators provided assistance services. A programme of technical assistance for schools was also made available. 69 Social development The main outcome indicators related to the social development dimension refer to (according Gigler, 2004) the access and the quality of basic social services, in particular in education and health as well as the access to social programs of the government. The previous sections have already shown that KEP has had a positive impact on all these dimensions, especially through the services provided at PAPIs. Within the project, a large number of training courses (e-learning or not) in various skills (not only in ICT) were provided to communities, both for the adult population and for the institutional representatives. Many of these courses were carried out in cooperation with foundations and NGOs from other communities. Also, PAPIs have delivered health related services, which have significantly contributed to improved knowledge about health practices, but also have increased access to better quality health services (using Internet), in their beneficiaries' opinion. In addition, within KEP, the access to e- government services have significantly increased and the knowledge about the governmental programs and policies have improved among local people. Nevertheless, with respect to the access of the poor to social government programs, the results are very limited as the available e-government services in this area are still insufficiently developed (some selective information and/or forms on a site, at most) for improving the efficiency, transparency and reach of these programs. However, from the local stakeholders' point of view, improved local schools represent the most significant benefit of the community from the project. The importance of educating the young in the use of ICT is well recognized today. The European Commission included the use of ICT in Education in the Digital Agenda for Europe (action area: enhancing digital literacy, skills and inclusion) and proposed a number of actions aimed to improve the educational outcomes.72 KEP included services as well as ICT equipment provision for the schools which are LCEN nodes. The services provided within the project consisted of: certified courses for using ICT for teachers and librarians; actions for the integration of ICT in education;73 e- vacanta competition;74 technical assistance for use of ICT in schools (aimed at promoting 72 To increase digital literacy, to mainstream eLearning in national policies, to contribute to the professional development of teachers and trainers, and to support the students to make the best use of ICT within education. 73 The objectives in education took into account three dimensions: curricular (innovation in school programs), pedagogical (innovation in the learning process) and organizational (structural innovation, roles and functions accomplished by the people involved in education). The development of public policies for ICT integration in the pre- universitary education system was the first goal that was achieved through a joint-work between the MCSI and Ministry of Education. http://www.mcsi.ro/Minister/Despre-MCSI/Unitatea-de-Politici-Publice/Documente-de-politici- publice/formular_pp_TIC. 74 The e-Holiday! (e-vacanta) competition aimed at strengthening the positive practices in extracurricular educational activities and at attracting new users in rural areas (parents, grandparents, entrepreneurs, local citizens). In the period 2006-2012, seven editions of e-Holiday! were developed. For more details see http://ecomunitate.ro/evacanta. 70 the usage of technical and digital content resources in the learning process, but also in community development activities by attracting funds and building inter-school partnerships). As a result, the schools from KEP area score well regarding the ICT infrastructure, connectivity, teachers’ specialization in ICT and access of students to the ICT equipments. The comparison with the non-KEP area shows a more nuanced picture. For this comparison, we developed a typology in line with a recent European study on ICT in education,75 which focuses not only on access to technology or simple use of ICT, but on the instruments and methods for enhancing the children's informational capabilities so that to make the best use of ICT within education. Thus, schools are classified into three e-types according to 14 criteria as follows:  An e-equipped school is a school that meets all the following criteria: (1) the school has maximum 15 students per computer; (2) more than 90% of the computers are connected to Internet; (3) the school has an online presence (i.e. web page); (4) the school has technical support (hardware and software).  An ICT-oriented school is a school with ICT policies that meets at least four of the following six criteria: (1) the school has a development strategy based on ICT; (2) the school has regulation regarding the responsible use of Internet; (3) the school has regulation regarding the plagiarism; (4) the school has regulation regarding the use of social media in teaching-learning purposes; (5) the school uses filters to prevent access to undesirable content; (6) the school organized courses regarding the identification and evaluation of online information.  An ICT-supportive school is a school that meets at least three of the following criteria: (1) all computers used by students are connected to Internet; (2) the school equipments are frequently used as teaching support; (3) the school equipments are frequently used for informatics lessons; (4) the students have access to the computers both during lessons and after the school program. Based on the above typology becomes apparent the highly positive impact of KEP at the level of local schools. Thus, in the KEP area, 18% of students go to an e-equipped school, 35% of students are taught in an ICT-oriented school and 39% benefit of an ICT- supportive school. The gap between the KEP schools and the non-KEP schools is considerable, especially with regard to orientation towards ICT policies and the support provided to students. In the control non-KEP communities only 10% of students go to an e-equipped school, virtually no student is protected within the school by specific ICT related policies, and only 22% learn in an ICT-supportive school. 75 European Schoolnet and the University of Liège (2013). 71 Figure 10. Distribution of schools and students by e-types in KEP and control non-KEP areas 45 % of schools  KEP rural 40 % of schools  Control  non�KEP rural 39 % of students  KEP rural 35 35 % of students  Control  non�KEP rural 30 25 22 20 18 15 10 10 5 20 15 33 0 34 30 0 0 E�equipped schools Schools with ICT policies ICT supportive schools Data: KEP Survey (January 2013). Schools: N = 85 for KEP rural and N = 20 for control non-KEP rural; Students: N = 25,705 for KEP rural and N = 6,801 for control non-KEP rural. The e-typology is based on 14 criteria. The project brought substantial benefits for the children enrolled in education, both in access and in use of ICT. The main source for the children from rural KEP area to learn to use ICT is the school. As demonstrated in a previous section, the access to computers and Internet in schools is the most efficient way to reduce the use divide between children from urban and rural areas, children from households with access at home and those geographically disadvantaged, Roma and children from other ethnic groups, poor and better-off children. The project has not succeeded to close the gap between Roma and Romanians regarding access to and usage of ICT, but Roma children are better than they would be if KEP would not have been implemented. However, for the children from the disadvantaged groups, such as Roma ethnics or poor families, the access and use of computers and Internet at school can lead to improvements of their life chances. Furthermore, the school-community relation has improved as result of the project activities. Participation of both pupils and teachers in extra-curricular activities increased in KEP localities during the years of the program implementation. In 2012, almost 43,000 pupils and almost 3,800 teachers were involved in 1,926 organized events (KANTOR, 2012). The participation of such events contributes to the development of the cultural life and involvement of more inhabitants, which is a characteristic of community development. 'Local people were reluctant at first, in the first few months. Luckily, young people are more attuned to the Internet, and it is via them that we managed to introduce it to the old folks.' (Mayor, Luncaviţa commune, Tulcea, June 2013) 'The KEP began in 2008. From the outset it has been a success thanks to the e-holiday project, during the summer holiday. All the children from the local school took part. There were activities in which parents took part also, and we introduced them to IT. 72 Members of the local community also took part. [...] We took part in a twinning project with a school from Belgium, in Marche en Famenne. Our children met theirs; they took photographs and then posted them on a blog. Via technology they also practiced their French, because they had to communicate.' (Teacher and School Director, Teiu commune, Dâmboviţa, June 2013) Economic development With respect to the economic development, KEP succeeded to empower communities to access new opportunities. Increased Internet use and proficiency have resulted in enhanced access to remittances by improving the communication with community members left abroad for work. The active labor market services, the marketing services for local products (including traditions and tourism), the online banking services, the assistance services regarding e-commerce, the support services for farmers regarding agricultural machinery and equipments at competitive prices have all contributed to enhancing meaningful use of Internet, which has improved significantly the access to information and knowledge with a real potential to improve the productive activities of the community. The services targeted to businesses and non-agricultural self-employed (information about funding opportunities and assistance in drafting applications) have contributed to the development of the local businesses, but also to the significant enhancement of the use of e-government services…For improving access to financial resources, KEP included a grants scheme (e-growth), 76 which had limited effects: due to the underdeveloped local businesses from KEP area, the number of application was so limited that the scope of the scheme was extended to authorized individuals, family associations and micro-companies from all over the country. Access to markets and commercialization of products has not been significantly improved, particularly for farmers, as it is limited by much broader socio-economic factors which cannot be tackled by the project. Nonetheless, within the project a large number of information events and business-related events were organized at PAPIs.77 According to the mayors and PAPI managers, enhanced capacity to mobilize resources from outside donors is another important outcome of the project, not so much in relation to entrepreneurs, but in relation to the local authorities. In most KEP communities, the local authorities together with PAPI, and with the project assistance, have applied for 76 The e-growth grants scheme was launched in 2009 through an extended awareness campaign. 77 For example in 2012, 146 communities (out of 229) reported information events with more than 6,400 participants (entrepreneurs and farmers) on the following topics: National Rural Development Program (PNDR), EU funds and programs, source of financing for agriculture, tourism and SME’s, Measure 141 from National Rural Development Program (PNDR) - Support for semi-subsistence farms, Meeting with Agency for Payments and Intervention in Agriculture (APIA) – informations about APIA subsidies and identification of land areas, Measure 112 from National Rural Development Program (PNDR) - Setting up young farmers, E-Antreprenor, Measure 121 from National Rural Development Program (PNDR) - Modernization of agricultural exploitations, Meeting with the County Office for Employment (AJOFM) and presentation of the EU programme for Human Resources Development (POSDRU), “Green House� Program, others. (KANTOR, 2012, p. 55) 73 projects funded from various sources, in particular for the creation or improvement of the local infrastructure (public buildings, roads, water supply, gas supply, etc.). These funds represent in many KEP communities the key source for financing economic, social and cultural activities, hence for community development. Table 2. Number of grant projects obtained by local authorities in KEP area by project stage 2011 2012 Set up projects Number of. 619 440 Number of communities reporting 181 142 Approved/Financed projects Number of. 316 259 Number of communities reporting 150 130 Implemented projects Number of. 158 143 Number of communities reporting 90 76 Source: KANTOR (2012, p. 72). One community may have one up to more than 20 projects. 62 communities (out of 229) have no project or no answer to their applications. In 2012 the total number of projects was smaller than in the previous years because there were less calls for proposals for EU-funded grants in most operational programmes targeting rural communities and local authorities, therefore there were less opportunities for project development compared with the period 2008-2010. 'We have focussed attention on one thing: European projects. We have applied for 19 projects worth 38 million Euros. An integrated project with five parts: drains, extending the water mains, road surfacing, an after-school project, and conserving and promoting cultural heritage.' (PAPI Manager, Prundeni commune, Vâlcea, June 2013) 'Luncaviţa has carried out major infrastructure projects and also smaller awareness projects. For example, access to mains water: projects worth more than 400,000 Euros. We’re one of the few villages in Dobrudja that has round-the-clock water all year round. We achieved this thanks to Knowledge Economy. We received funding for the majority of projects. It’s more than simple: you apply online.' (Mayor, Luncaviţa commune, Tulcea, June 2013) Political participation The political participation of communities has improved within KEP mainly due to the development of e-government services, which has resulted in enhanced and more transparent information flow. As we have already mentioned, KEP provided support to all communities willing to create their own website. As a result, many KEP communities have their own website besides the information available on www.ecomunitate.ro. This is a key outcome indicator for the community empowerment (Rogerson and Itoh, 1998 apud Gigler, 2004). Table 3. Online consulting initiatives of local authorities from KEP area 2011 2012 Community website Number of. 929 413 Number of communities reporting 124 112 Blogs Number of. 210 150 Number of communities reporting 50 57 Online local platforms Number of. 172 118 (developed for networking) Number of communities reporting 59 51 Source: KANTOR (2012, p. 69). 74 The capacity of local authorities to organize online communication and consultations has increased considerably during the project. In approximately two thirds of the KEP communities, local authorities are developing and improving web sites, blogs, and forums. In many communities, the PAPI staff is responsible for creating and updating the websites of the communities or those of local authorities. Table 4. Participation of citizens in the consultation processes in KEP area 2011 2012 Answers received from inhabitants Number of. 1,876 1,539 for town hall announcements Number of communities reporting 82 80 Questions asked by inhabitants Number of. 2,083 1,783 Number of communities reporting 90 84 Other comments received Number of. 1,828 1,582 Number of communities reporting 77 74 Source: KANTOR (2012, p. 67). In addition, the enhanced informational capabilities of population resulted in increased community participation in collective actions and also in enhanced ability to influence the local decision making processes. Thus, the participation of citizens in consultation processes increased in the first years of the project, after which it stabilized. The level of participation in online communication with local authorities is rather weak (table 10), but most KEP communities are small rural communities, territorially compact, with strong social networks, in which direct communication is the dominant practice. Online communication is seen as a major need in relation to the migrants abroad or to the people moved to a city, but is not favored in relation to people 'next to your door'. Under these circumstances, most probably the rates of online communication with the local authorities are not going to increase significantly in the future either. As a solution, the local authorities from some KEP communities issue a local newspaper or a newsletter which is distributed to various institutions and to citizens and also posted on the official website. The project promoted knowledge sharing activities, which increased the capacity of KEP communities to coordinate actions with other communities. In this sense, 40 exchange visits were organized between KEP communities, in which representatives from all 255 KEP communities participated. In addition, five international study visits in Ireland, Hungary, Lithuania, Denmark and France were realized with the participation of representatives from 52 KEP communities.78 78 Information from www.ecomunitate.ro 75 Cultural identity In the cultural area, KEP provided through LCENs (at PAPI) assistance services both to individuals and to communities for creating their own website. In addition, the project portal www.ecomunitate.ro has posted material about each of the 255 covered communities. In this way, the knowledge-disadvantaged communities selected within the project benefited of support in creating a space to disseminate their own content and to share some of their experiences with other communities and the public in general. This new opportunity fostered the development of many cultural related activities, mostly in the area of cultural heritage and tradition revival. 'Through the activities we carry on, especially the library projects, we aim to revitalise our traditions, to enhance the value of our local cultural heritage. In 2008, we implemented the monograph project , purchasing an IT application to manage the books in the library and creating the village library website. [...] In 2011, we created this historical guide with volunteers from the community, a large number of them being PAPI Saschiz Volunteers. The historical guide also includes a multimedia DVD presenting Saschiz, as well as games relating to the information in the textbook.' (PAPI Manager, Saschiz commune, Mureş, June 2013) 'We’ve revived the tradition of craft items made from rushes and fabrics. Our local artisans taught the children to make them, too. We take part in international fairs. This year we went to Vienna. We revived the tradition thanks to Knowledge Economy.' (PAPI Manager, Luncaviţa commune, Tulcea, June 2013) Two nodes played a major role in this area, the public library and PAPI, which in many communities functioned as 'social hub' hosting various community activities unrelated to the actual use of ICT, from community events, trainings, fairs, medical visits to weddings. PAPI - Role of intermediary The KEP was designed as an ICT intermediary 79 so that to foster the local technical appropriation and community ownership….The central management team of the project played a much stronger role at the outset of the project and has gradually transferred the technical and management skills to the local communities. So, the local communities have gradually appropriated the technology and gained ownership of the project. Out of the 255 communities selected initially in the project, 229 (or 90%) communities have prepared a sustainability plan for the LCEN and are still functional. PAPI is the central node of LCEN, by design, and it is the main provider of services within community. As services have been tailored to the community needs, they differ from one community to another. In 2012, the number of services offered per community varied from 1 to 22, with an average number of 14 services (KANTOR, 2012). 79 An ICT intermediary is defined as a person or organization providing effective support to local communities in the use and adaptation of technology. (Gigler, 2004) 76 Figure 11. Number of communities offering different services in KEP area, 2012 Services  for inhabitants Services  for antrepreneurs 192 183 186 169 179 156 150 138 131 124 123 126 120 109 107 99 106 86 74 65 43 50 173 160 168 142 149 148 89 26 104 107 90 99 85 108 46 57 115 107 113 76 90 104 ce ... ea on ns s ns t s et  tr ts es er t f g en g er  o ing CD ce g ng io  ar m ling yin tin o io ism en iti rn t rm an sin m u sit in itm ot vi at ad at bo  in in er iti e  te v ul om op eb Sc Pr  fa Ph r rti om ol ic to ze e ic se es ra rit Ed th ns  In s a ru un R t w oc un ve W or t �c e  co c  to   se ur ICT m ot r e Ad  re ar in ani t f or T  es m Ph w ss ur or fo pp IC ur g ce in rd co pp Or bo e  s Su  c us Ha Ac e  nc g  Su sic La r b lin  co ta as ot On fo sis ng Cl om n  As ni io Pr ai at Tr rm fo In Source: KANTOR (2012, p. 22). The visibility of the PAPIs within the communities is high, as half of the population declared that they know about it. However, in the rural areas more people heard about and went to PAPI (57% and 33%, respectively) compared with urban areas (41% and 20%). The centers have managed to answer the population’s needs through the offered services. The majority of those who went to a PAPI with a problem declare to have solved it (71%, with no significant differences between rural and urban localities). Therefore the global satisfaction is very high among the people who benefited of any services or just attended the events organized at the PAPI centers. A basic condition for the PAPIs to respond to the beneficiaries’ needs is to be well equipped in terms of ICT infrastructure, connectivity to the Internet, human resources and premises….The PAPIs delivered a broad range of services to communities. Overall, about 300 thousands persons declared to have benefited of one or more PAPI services. About 70% of beneficiaries are from rural areas. More than 70% of beneficiaries declare to be (very) satisfied with regard to nearly all services. Conclusions and recommendations Following the ICT impact chain: the access to information and ICT has been significantly improved; the basic ICT skills were ensured; the higher Internet proficiency accompanied by guidance and support provided through PAPI services has resulted in meaningful use of Internet, which led to enhanced informational capabilities; and thus the project has had a positive impact on well-being, both at the individual and community level. The multiplier effect of informational capabilities varies from one area of life to another. Positive impact has been highlighted in the informational, social and, especially, in the cultural areas. Less extended impact was observed in the economic (income generating) and vertical communication (eGovernment) areas, however high in relative terms (compared to the national average), particularly taking into consideration that the impact in these areas is limited by broader socio-economic factors found out of the scope of the project. 77 Services provided through LCENs (at PAPI) played the most important role in enhancing people's informational capabilities and in increasing the odds of meaningful use of Internet in order to derive benefits in the social, economic, political and cultural areas. The project played the role of an ICT and social intermediary. A rather successful process of technical and social appropriation took place at the local level. The prospectives for the future seem optimistic, according to the mayors of the participating communities. Nonetheless, sustainability remains an issue to be considered. 78 Annex 6. Stakeholder Workshop Report and Results Not applicable. 79 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR [Excerpts from the Borrower’s evaluation report] ASSESSMENT OF OUTCOMES Relevance of Objectives, Design and Implementation The project and its activities were highly relevant not only to ‘national’ priorities, but also to international concerns relating to sustainable development and good governance in the e- government services relation. The consistence and coherence of the proposed objectives and measures were logically linked to the socio-economic analysis, mutually consistent and were well embedded in the ‘national’ MCTI targets for small communities. The expected and unexpected utility effects were realistic and globally satisfactory in the context of wider social, environmental and economic needs of small communities. The effects obtained in the working programmes show sustainability, even after the end of the KEP no further public funding are allocated. The development of an early education system for small communities should help avoid the socio-economic losses within services. Improved information society services, inter-ministry coordination has socio-economic implications through improved efficiency of government. Improved information society policy and the adoption of ICT methods would have a major socio- economic impact for Romanian small communities. Relating to ‘Overall relevance’, the KBE projects are considered “Highly relevant� to be sustainable. KBE has been very successful in achieving a measure of institutional sustainability and a medium term financial sustainability to ensure continued development of the KEP over the next years, fully supporting the National and European Digital Agenda. The principal risk to sustainability is that the recent closure of the KBE programme and particularly closure of the PMU affect Local Centres which have been so important in driving the programme, will result in a substantial slow-down of momentum across all outcomes. Whether this happens will depend on the local, key government staff and the individuals selected for key posts in the LCeNs. Another significant risk is the rapid turnover of ‘LCeNs’ personnel in private sector employment. The rating of project implementation approach was “High Satisfactory� (on a scale from unsatisfactory to highly satisfactory). While project coordination was excellent, the full effectiveness of project implementation was a little difficult by the approach involving numerous subprojects which were designed only after project inception, which imposed a major responsibility on project coordination and working in a crisis time. Efficiency Project management was efficient, effective and responsive. Management by the PMU received particular praise. Financial planning and management appear to have been effective and prudent, with no instances of serious overspend and a project that had the necessary resources for a 4 month extension without additional cost. Efficiency of the resources allocated with the respect to the outputs and results show that activities within the KEP Project on Technical Assistance, conducted jointly by schools, local libraries, allowed the conclusions and recommendations that can be the basis of a new approach at national level relationship between these institutions in communities. May be mentioned: (i) Implementing a wider and more effective use of ICT; (ii) Effectiveness in organizations; (iii) More effective exploitation of ICT in training; and (iv) Effective use of "network" already established. 80 In terms of Cost-Benefit the project activities helped in attracting funding for communities’ development of more than EUR 250 million. Considering that only one part of the 255 communities accessed these funds and entire KEP used about US$41 million allocated, resulting impact factor is bigger than 6:1. Efficacy The impact assessment conducted by the World Bank between May-June 2012 in the project communities (KEP) shows important development values, equalling or exceeding national and rural area averages along all dimensions of the ICT impact chain. - The initially KEP target: 200 communities were achieved by 255 knowledge disadvantaged communities (208 communes and 47 small cities) from 1500 applications; - the percentage of individuals that use regularly the Internet is 41% in the targeted communities, almost equalling the national average of 43% (that includes large urban areas), while; - the percentage of individuals that used e-government services is of 12%, much higher than national averages of 7% (or of the control group of lower than 4%); - in the targeted communities with no access to Internet before the project, 46% of the relevant population is now connected and use Internet, as opposed to none before the project start; - 43% of relevant population are satisfied with the results brought by the project, higher that the target of 40%; - regularity of using the Internet ( percentage of population that used every day) 41% versus the target 28%; - population that used eGovernment services, 12% vs. the target 11%. Initial goals were met and exceeded in most of cases, often KEP communities indicators are better than national averages since these are aspects considered along the 7 years of implementation. Project Development Objective indicators targets were exceeded. 229 communities operate efficiently and the level of activity is sustainable in the future. 43% of the relevant population in these communities have used LCeNs and is satisfied. Regularity of using internet exceeded the 28% target, now to 41%. Also the quality of ICT as demonstrated by the use of Internet for the management of official forms reached 12%. Sustainability of LCeNs increased to 30% and so is fully supporting National and European Digital Agenda strategies. The project has identified and developed new targets that are one of the keys to its success. All, exceeds national averages performance, work as driving factors and must be mentioned that the project helped to attract other additional funding for communities’ development, of more than EUR 250 million. Comments on Impact Assuming the World Bank’s “Assessment of the Outcomes�, “The project contributed to accelerating the participation of the targeted 255 knowledge disadvantaged communities in the knowledge-based society and economy in Romania to a satisfactory extent with substantial efficiency�. Most of these communities operate satisfactorily towards their sustainable future in line with their own business plans contributes to the project impact. Further challenges for this impact evaluation have been:  Indicators that permits evaluation of relevance, implementation performance and success of the project.  Implementation of hundreds of equipments, acquisitions, projects and related contracts, activities promotion, training, competitions and financing programs. 81 The EPSA 2011 Awards at the „Smart Public Service in a Cold Economic Climate� Best Practice distinction rewards certifies the outstanding results that KEP Project has had until now. “The capitalization of a community’s cultural memory can generate the local sustainable development impact. e-Culture has intervened at the level of local libraries in order to support the individual development process of children, adults and senior citizens and to offer them access to local values by means of modern cultural services. The main impact of KEP was given by the developed projects in the four area of activity. The very important impact role of KEP in KBE was played by more projects and related events, conferences, all sustained by the Portal in area of: Local Administration Projects; - http://www.ecomunitate.ro/Proiecte_Administratie_locala(26397).html Educational Projects; - http://www.ecomunitate.ro/Proiecte_educationale(26394).html - eVacanta (eHoliday); http://www.ecomunitate.ro/Proiect_e-Vacanta(26396).html (2008- 2012) Cultural Projects; - e-Culture. http://www.ecomunitate.ro/e-Culture(24556).html Business Projects: - eCrestere (e Growth); http://www.ecomunitate.ro/ecrestere - eAntreprenor ; http://www.ecomunitate.ro/Proiect_eAntreprenor(26395).html Promotional materials was addressed in dozens of models in thousands of copies, dozens of conferences and events. Detailing the impact of the 4 areas of action can remember on: Local Public Administration-LPA The Technical Assistance for the Development of the LCEN Sustainability Plans was oriented to obtain from the start a higher impact:  Assistance provided to the LCEN staff,  Qualitative analysis of the information included in the sustainability plans, o Types of funding o Services regarded as a priority in ensuring sustainability  Developing an interaction and networking program for the LCEN staff,  The LCEN staff assessment,  Support for sustainable development of local communities,  Facilitating partnerships between KBE communities and private organizations. A very important indicator for the all around evaluation of the KEP impact in communities is the indicator L1 – “Services provided by the different LCeN nodes�. The decrease with impact for some of them, example: the average monthly number of PAPI users falling from 110 in 2011 to 96 in 2012, means that this evolution is by reason of the spreading of internet access in communities. Less than 5 percent of KEP communities have for the network services the sole provider of broadband internet. eCulture and e-Education One of the most important impact was given by e-Culture programme. What do the participating libraries in the projects, now offer to their community members:  A framework for designing local groups of initiative;  Access to information on writing educational, cultural and local development projects, based on the community’s tailored needs; 82  A way to identify funding sources and partners to carry out projects of any kind;  Assistance in identifying, preserving and protecting local cultural heritage and promoting it at European level;  Access to high speed Internet and different sources of information;  Introduction to electronic books and eReaders as a modern alternative to classic lecturing;  Digital literacy and IT skills development services;  Access to modern digital services - eLearning, Internet banking etc.;  Easy access to employment opportunities and professional retraining. In this way, “print culture� is complemented by the “digital culture�, enabling consumers of information to simultaneously benefit by the same cultural and educational resource.� Of a particular importance is the large number of projects initiated and developed in schools which reflect a clear image of the impact achieved through this programme in schools. List of printed publication, addressed to all units in secondary education and educational partners in Romania with the highest impact include 15000 Manuals and Guides, Brochures and Promotional materials. The Manuals and Guides addressed to all Romanian secondary education units:  Guide for Teachers “Utilizarea calculatorului personal:aplicare TIC in scoala si afaceri, 527 pages, 15000 copies,;  Guide for Schools; “Inovatie si bune practici in educatie prin resurse digitale�, 96 pages, 15000 copies,  Guide for Schools “Integrarea noilor tehnologii in educatie�, 102 pages, 15000 copies;  Manual “Utilizarea TIC pentru activitati educationale�, 500 pages, 1000 copies;  Manual “Diseminare al bunelor practici din proiectul „Asistenţă Tehnică pentru Utilizarea TIC în Şcoli şi Biblioteci Locale�, 96 pages, 2000 copies. Brochures:  DIGITAL LIBRARY eCULTURA, Programme for innovation in libraries, 28 pages, 1000 copies. e-Business The impact on companies and businesses area was done in several ways. Especially the component 3 “Promotion of e-commerce and innovation support for MSMEs� with 2 subcomponents had a big impact, indicators being presented in evaluation Chapters:  Subcomponent 3.1 Portal for promotion of e-commerce and business networking  Subcomponent 3.2 Grant Facility Grant facility for “Micro, Small and Medium Enterprises� was oriented to investment projects that were finalised, were allocated funds through the following specifications:  Tangible assets (technological and ICT equipments): 42%  Intangible assets (brevets and Software and ICT applications license: 14%  IT Services: software solutions: 16%  Consultancy services (business plans, implementation advisory, market analysis) : 12%  Training services: 11%  Marketing services (Trades/fairs/ conferences attendance, Web shops, Website, Marketing online): 2%  Audit services: 2%. The impact was in multiple forms. The competition “e-business entrepreneur� - eAntreprenor , (July-September 2012), aimed at creating, supporting, strengthening and promoting an 83 entrepreneurial culture in rural areas and small urban capable of accelerating the initiation of successful business and increase competitiveness locally. Qualitative perspective on KEP achievements KEP project implementation generated a series of changes in small communities and these changes are well appreciated by the inhabitants. There was a unanimous opinion that the program is beneficial to the communities and the infrastructure represents the backbone of any future achievement based on the shared knowledge. There are features that can easily be listed as common achievements and KANTOR evaluation reports gives us numerous and detailed results. In the first two years 2007 and 2008, main actions of the KEP project have been to provide the ICT infrastructure which represents the backbone of any future achievement. The purpose of this evaluation report was to analyze and evaluate the results obtained at the end of project in comparison with the previous years, using qualitative perspectives. Qualitative perspective on KEP achievements and the Sustainability plans share many elements reflected in the big number of positives results. Local Public Administration-LPA PAPI has been the central KEP implementation point for more qualitative reasons; addressed to all categories and ages of inhabitants, only new established centres, with sustainability of the node ensured for many years. Today it is not now possible to imagine a town hall, a school, a library, people not using ICT in the future. In addition to this fact, KEP is a step in advance for national strategy for development of e-government, will further increase the adoption of ICT and its benefits in society and governmental institutions. The impact can be measured in many ways:  122 communities submitted 221 projects whose direct beneficiaries are the LPAs;  65 with 1 project;  35 communities with 2 projects;  12 communities with 3 projects;  8 communities with 4 projects;  1 community with 5 projects. In conclusion 58 communities submitted 591 projects whose direct beneficiaries are farmers, individuals, the number of projects varying between 5 and 125 projects/community. Sustainability plans have been assessed both in terms of funding and in terms of priority of services for sustainability. Types of funding in the financial analysis has led, at a first stage, to the following findings:  38% of the communities (96) benefit 100% from LPA (local public administration) funding;  33% of the communities (84) benefit between 70-90% from LPA funding;  16% of the communities (42) benefit between 50-70% from LPA funding;  13% of the communities (33) benefit less than 50% from LPA funding. The next stage in the analysis, in terms of priority, has focused on identifying other resources which may be taken into account by communities in order to ensure co-funding for LCEN activities. e-Education From a qualitative perspective are the obtained results in ‘Training the LCEN staff’ with a view to develop PAPI services through ITC:  103 LCEN administrators were trained in Web Development and ITC tools applied in business who carried out the following: o 103 web sites, 84 o 103 TA reports which will comprise:  The identification of a software application for business development;  Market research for services and products diversification;  The identification of an online promotion of the services provided. At level of schools there were initiated 43% more projects that use external sources of financing such as other programmes. This is a clear measure of the impact that Technical Assistance for schools had in 2012 in KEP communities.The percentage of pupils using computers in schools maintained a constant evolution and over 2% more teachers are using computers in school. Very encouraging for 2012 is the percentage of teachers which are using computers in schools for added value activities such as the teaching process, preparation of school projects and preparation of lessons plans. For each of these examples we observed an increase of around 5% at the level of all teachers, a development clearly supported by the Technical Assistance for schools. e-Culture The qualitative improvements in e-skills for using ICT in cultural activities meant creating of:  Websites and blogs to promote the cultural activities and the services provided by the library;  Newsletters and digital magazines;  Movies, audio recordings, photo montages, presentations, promotional materials in digital format;  ICT courses on specific subjects, addressed to various categories of population;  Library databases, digital book repositories;  Digitization of books and photographs for the Europeana cultural heritage. A list of activities and projects underlines the importance to the library as public institution:  documentaries and multimedia presentations for preserving and promoting the multicultural and multiethnic local values,  reading and audio sessions were organized, during which a series of literary writers of different nationalities were studied,  cultural items of local heritage were selected and recommended for exposure in the Europeana portal. A list of cultural projects underlines the large number of developed activities and the qualitative improving of socio-economic targets:  “Library - Centre of Information Spreading�,  “Tradition and Art: Authentic Transylvanian Saxon Fasching�,  “Memory, tradition, heritage — A resource for the future�,  “Wings Rustle� Theatre Club�,  “eReading, eTraditions — Link Between Generations�,  “eDialogue – Culture without Barriers�,  “Digital Cultural Bridge through eLibrary and eReaders Services�,  “Culture by ICT Knowledge�,  “Reading as a gift�,  “Rural economy based on knowledge — eGospodari�,  “Traditional, balanced and healthy nutrition�,  “The library, promoter of the tourism development and of the crafting ntrepreneurship�,  “eLibrary & Virtual Technology�,  “Tradition and Innovation in Your Library� as a Book Club for Senior Citizens. 85 e-Culture created opportunities for various categories of beneficiaries, developed the local volunteering trend and created the qualitative context for other successful projects, to sustain quantitative results. e-Business The preponderant businesses in KEP communities are specific to the subsistence agriculture type. Local farmers are mostly individuals and only a small number of registered SMEs are active. As a result, the surveys had to be adapted to meet new requirements in assessing the impact of the KEP project over the e-business component. In the development of the business sector a principal role was played by PAPIs. From this perspective, the e-learning and e-business indicators can be correlated. In the Grant facility for Micro, Small and Medium Enterprises, most of the expenses were directed to the acquisition of the tangible assets, representing technological equipment and ICT equipment. From those, approximately 62% were ICT equipment. The program results analysis presents the higher impact in the communities: Programme's results analysis  (no. of goods/ services) 700 No. of Technological equipments No. of ITC equipments 600 No. of brevets 500 No. of Software  and ITC applications licenses No. of IT Services:  software  solutions 400 No. of Consultancy services 300 No. of training courses No. of people  trained 200 No. of Trades/faires/conferences 100 No. of webshops No. of websites 0 No. of advertisement Budgeted Realized Figure 12: e-Business program results – no. goods/services/activities Approximately 85% of the beneficiaries activate in the service sector, from which 32% in the commerce sector. There are several explanations that explain this situation: (i) E-business solutions apply best to service dedicated business; (ii) As for the industry companies, IT solutions came along with technological automation, so they will be included in the value of technological equipment; (iii) At present, given the agriculture sector’s structure, e-business solutions, online marketing and ICT infrastructure are not that useful; (iv) The commerce sector is not financed through many no reimbursable programmes; and (v) The weak presence of the companies from industry and agriculture was caused by the following the facts: the investments are oriented through technological equipment. 86 Distribution of beneficiaries by activity fields 5% 11% Industry 32% Services Commerce Agriculture 52% Figure 13: Beneficiaries distribution in e-Business programme The distribution of beneficiaries shows the shared impact in the economy sectors, qualitative changes of the structure from the agriculture initially stage. Impact at the level of the local communities After 2 years of TA, a positive evolution has been noticed in all the communities and especially in relation to the shift of mindset and approach in the ITC daily use. The impact obtained by the KBE TA at the level of the local communities:  250 communities benefitted from training courses in the field of financing request draft, of which;  250 communities drafted a project related to a financing request;  180 communities (more than 70%) managed to submit financing request declared eligible. The total value related to these projects is more than 240 million Euros:  103 communities were trained to use ITC in the business development;  50 websites were created for local traders;  206 communities concluded at least one public-private partnership. Between types of financed activities we have:  Consolidation and diversification of PAPI services (18 projects),  The LCEN staff training (12 projects),  -Volunteer based work (1 project). More than 85 communities were involved in the implementation of 31 projects financed via the CONECT grant scheme which led to the following results:  The organization of 31 training courses, of which, 19 received NATB certification  More than 630 persons, both LCEN members and other persons in the KBE communities participated in the training courses, of which, more than 360 received NATB (CNFPA) certification; Thus, the level of satisfaction regarding the TA KBE services, registered 89% as compared to 62% level of benefits perceived by the communities in the beginning of the project. Also, the professional satisfaction of the LCEN staff was extremely positive in 80% of the communities. Through the numerous and complex activities planned, the PMU team played an important role in the increase of the level of satisfaction and trust in the services provided. One can mention the higher interoperability between programs and activities, the strong impact in communities, in population, in companies. 87 ASSESSMENT OF RISK TO DEVELOPMENT OUTCOME Assesses and rates of “Risk Development Outcome� of the operation presents the risks at the time of evaluation affecting in maintaining or expected outcomes. Overall risk can be rated80 as “Moderate�. A main risk is represented by the communities’ incapacities to ensure future co- financing or the human resources needed for future projects’ implementation. The project extension of 18 months after the original closing date made possible the implementation of risk mitigation measures identified by mid-term review, as the intensified to communities and schools that increased the impact and the sustainability of the project. Risks are internal, in administrative area, or technical and specific to each operation. The Rating helps to identify that requires priority or more attention. In plane of social, political and institutional the risk are exceeded and maybe, only in some points, so that risk can be assessed as moderate. The first risk at the final of project is to maintain the outcomes. The actual sustainability of the project impact is significant, 90% at the level of the communities assisted partially by the project. 229 communities from 255 communities initially supported by the project, implement measures in line with their approved sustainability plans. 30% of the communities have concluded partnerships and are implementing various development projects. Moreover, 51% of communities have projects pending signing. The evaluation of sustainability indicators demonstrated that 229 communities had the best sustainability premises and 26 communities were at risk regarding the sustainability. The results above risks and regarding the positive or negative evolution of the LCEN performance lead to the following findings: (i) More than 60% of the communities have had a steady evolution regarding the quality of their activities and services; (ii) The positive evolution recorded by 40 communities is mainly due to the positive results recorded in terms of number of projects declared eligible. The best recommendation that can be made in this case is for the Town Halls to find solutions for overcoming obstacles and keeping the PAPI nodes opened as much as possible. Perhaps more important than this, is to assure consistency by attracting and maintaining a long time relationship with their PAPI manager. An alternative to keeping the PAPI open is to give these responsibilities to the librarian. Unfortunately, this is only possible in a small number of cases, as in general the same communities having difficulties and associated risks keeping the PAPI node functional state similar issues for the library: personnel is hard to find, there are no available funds or there is a lack of facilities /locations for them. ASSESSMENT OF BANK AND BORROWER PERFORMANCE Assesses and rates of Bank and Borrower performance are in context with provision of funds, equipment, consultants' services, ‘monitoring and evaluation’ and training to public authorities for the development of an integrated network for citizens status improvement in information and documentation. 80 Rating scale Negligible to Low, Moderate, Significant, High 88 Bank Performance Services provided by Bank ensured the capacity at entry for all operation and supported effective implementation, ensuring quality supervision for regular operation inclusive activities after loan closing. Quality at Entry. The Bank provided to KEP funds for the following nodes and supported activities: (i) the basic and lower secondary schools; (ii) the local administration offices (Town Halls); (iii) the public libraries; and (iv) Public Information Access Points (PIAPs). The quality at entry, take into account all operations, the strategic relevance and approach, technical, financial, economic aspects, also policy and institutional aspects or monitoring and evaluation arrangements. Risk assessment being ‘Satisfactory’ and no other shortcomings can be mentioned, so we can evaluate ‘Satisfactory’ quality at enter. Quality of Bank supervision. Work Bank activity, from the start of the project, was resumed at supervision for all operations in proactive identification of opportunities for development and resolution of appeared threats. The quality of Bank, inclusive activities after loan closing, because were no shortcomings can be qualified Satisfactory. Rating of Overall Bank Performance. Taking into account Bank performance in Assuring quality at enter and Quality of supervision, and outcomes ratings, the Bank performance was for both dimension rated ‘Satisfactory’. Borrower Performance The Government of Romania, through the Ministry of Communications and Information Technology (MCIT) in 2005 has received the loan from the World Bank for financing the Knowledge Economy Project. The funds were split between more distinct uses such as Acquisition of Goods, Operational Costs, Training and Monitoring & Evaluation. According to the Loan Agreement, the objective of the project was to support the Borrower in its efforts to provide access to ITC services and improve efficiency of the public system to prepare Romanian's people for the knowledge society and economy. Overall Borrower Performance can be treated and assumed between Government and PMU Performance. The first responsibility remain in charge of Government because the fund allocation was a critique element than influenced the project behaviour. Government Performance. The Borrower, including the government (MSI), ensured quality of preparation and implementation from appraisal towards achievement of development outcomes for all four components. The impact assessment conducted by the World Bank in May- June 2012 in the project communities (KEP) shows important development impact, equalling or exceeding national and rural area averages along all dimensions of the ICT impact chain. Mentioning minor shortcomings in government performance, the rating can be qualified Highly Satisfactory. The response of the project to all components is considered completely satisfactory. PMU Performance. The PMU has adequate internal controls for the project, including regular reconciliation of bank accounts, adequate segregation of duties and performs monthly reconciliation of disbursement summaries of the World Bank with project accounting records. The PMU uses a comprehensive set of accounting policies and internal control procedures in accordance with the Romanian legislation and the project financial management practice. The allocation of counterpart funding was timely. 89 Rating of Overall Borrower Performance. The rating for PMU can be qualified to Highly Satisfactory taking in account the achievements, impact and numerous awards. Taking into account Borrower performance for both dimension (Government and PMU), and judged in terms of outcomes ratings of the KEP project, the Borrower Overall Rating was rated ‘Highly Satisfactory’. Finally it must be mentioned the small number of PMU staff along the project life, an average of ten, now remaining 6 persons. Selected manuals and Guides prepared under the project: 90 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 91 Annex 9. List of Supporting Documents Project Appraisal Document; Loan Agreement and amendments; Project Restructuring Papers; Aide-Memoires, Implementation Support Reports and Management letters Project M&E Reports: LRDP KANTOR Ltd. Monitoring Reports (16 Quarterly, 8 Evaluation, 4 Exhaustive, Final Reports and 4 Annexes 8th Evaluation, 6 Case impact studies, 4th Six case studies); M&E report for Grant Facility M&E documents (Final Evaluation Report, TOR, working and ranking files); Households and Institutional Impact Assessment Country Partnership Strategy and Progress Report on the Country Partnership Strategy for Romania (FY 09-FY13); National Information Strategy; Partnership Agreement for Romania 2014-2020 – draft; Lisbon Strategy and eEurope 2005, i2010 and Digital Agenda 2020; eComunitate website. 92 IBRD 40252 20°E 22°E 24°E 26°E 28°E To ROMANIA Uzhhorod To Ivano-Frankivs'k KNOWLEDGE ECONOMY PROJECT U KR AI NE PROJECT SITES COUNTY (JUDET) CAPITALS Hudesti Cristinesti NATIONAL CAPITAL ROMANIA 48°N Draguseni 48°N MAIN ROADS Halmeu Gramesti Galanesti Vlasinesti Vicovu de Sus Candesti RAILROADS Balcauti Braesti Satu Mare Sucevita Calafindesti Iaslovat BOTOŞANI COUNTY (JUDET) AND MUNICIPALITY Tautil-Magheraus Poienile Izei Moldova-Slita Arbore Siminicea Botoşani (MUNICIPIU) BOUNDARIES SATU MARE Baia Mare Borsa Frumosu Cacica Suceava Copalau INTERNATIONAL BOUNDARIES MARAMUREŞ SUCEAVA Fantanele Berchisesti Prajeni Simian Arinis Valea Moldovei Plugari Targu Lapus Dolhasca Cherechiu Parva Ostra Sacueni Marghita Camar Carastelec Dobrin Rus Vima Mica BISTRIŢA- Poiana Stampei Draguseni IAŞI 30°E To Chislaz Saru Dornei Panaci Brusturi Cristesti Costesti Victoria Diosig Budapest Zalău Simisna NĂSĂUD Targu Frumos Iasi To Spinus Sinteu Crasna Bobalna Bistriţa NEAMŢ Pastraveni SĂLAJDragu Sinesti Budapest Oradea Hangu Negresti To Cornesti Chisinau H UNGARY HUNGARY BIHOR CLUJ Sieut Tulghes Bicaz Piatra-Neamţ Scheia Lazareni Sanpaul Valeni Catina Tarcau Rebricea Cluj-Napoca Bicazu Valea Ursului Holod Suatu MUREŞ Remetea Ardelean Rafaila Feresti Arsura M O L D O VA Budureasa Frata Sarmasu Gornesti Joseni Gheorgheni Buhusi VASLUI Vaslui Marisel Târgu Mureş Ciumani Balan Traian Lunca Banului HARGHITA Miercurea Pilu Zerind Apateu Baisoara Viisoara Iara Sanger Nirajului Palanca Bacău Oltenesti Hoceni Ludus Ghindari Varsag Pasareni Corund Tamasi Bogdanita Garda de Sus Sanpaul Dangeorgiu Vetrisoaia To Subotica Campeni Iernut Fantanele Miercurea-Cuic Vlahita Santimbru BACĂU Ibanesti Iana Falciu ARAD ALBA Murgeni Arad Zlatna Laslea Saschiz Homocea Alba Lulia Valea Viilor Pochidia Cavadinesti COVASNA Varadia de Mures Rupea 46°N 46°N Ilia Geoagiu SIBIU Belin Movilita Marasesti U KR AI NE TIMIŞ Bethausen Deva Sercaia Parau Sfântu-Gheorghe Tulnici Valea Sarii Targu Bujor GALAŢI Recas Saliste Nistoresti Jimbolia Sibiu Timişoara Barna Lelese Calan Recea Sinca Veche VRANCEA Focşani Grivita Liebling HUNEDOARA BRAŞOV Braşov Vintileasca Tudor Salasu de Sus Chiojdeni Vladimirescu Tormac Predeal Slobozia Bradului Schela Petrila Ghilad Farliug Baru Galaţi Giera Uricani Aninoasa Albestii de Muscel BUZĂU Balta Alba Grindu Reşiţa Patarlagele Brăila VÂLCEA Florica Jijila Luncavita Ceatalchioi Tulcea CARAŞ-SEVERIN Teregova GORJ Vaideeni Râmnicu Barbuletu Rau Alb PRAHOVA Buzău Nufaru ARGEŞ Ianca Mircea Voda TULCEA Pietrari Virfuri Bordei Verde Bestepe Anina Vâlcea To Novi Sad Oravita Târgu Jiu Albeni Bustuchin Danicei Vulcana-Bai Târgovişte Ploiesti BRĂILA Peceneaga Lapusnicu Dalboset Targu Carbunesti Ciucurova Licurici Stoilesti Mare Bania Gradistea Teiu Darmanesti Topolog Piteşti Marasu Sopotu Nou Glavile Amarasti Prundeni DÂMBOVIŢA Jurilovca Drobeta-Turnu Severin Creteni Balcesti Stefanesti Tatulesti Barasti Visina Produlesti ILFOV IALOMIŢA Tandarei Harsova SERBIA Filiasi Bolintin-Vale BUCUREŞTI Slobozia MEHEDINŢI Scaesti Slatina Roata de Jos BUCHAREST Lehliu Gara Perisoru Sefan Cel Mare CONSTANŢA Craiova Soparlita OLT Barza Ciolanesti Tatarastii de Jos Sarulesti Gurbanesti Vlad Tepes Borcea Stefan Voda Black Talpa Vasilati Plenita Dobrun Izvoarele Necsesti Didesti Letca Noua Botoroaga Varasti CĂLĂRAŞI Călăraşi Constanţa Sea Perisor Herasti DOLJ GIURGIU 44°N Urzicuta TELEORMAN Baneasa 44°N Alexandria Giurgiu 0 50 100 Miles Corabia 0 50 100 Kilometers Zimnicea B U LG AR I A This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other information To To shown on this map do not imply, on the part of The World Bank Sofiya To Varna Group, any judgment on the legal status of any territory, or any To Shumen endorsement or acceptance of such boundaries. 22°E 24°E Veliko Turnovo ˘ 26°E 28°E 30°E AUGUST 2013