48032 v2 IMPROVING DEVELOPMENT RESULTS THROUGH EXCELLENCE IN EVALUATION Review of IDA Internal Controls An Evaluation of Management’s Assessment and the IAD Review Report on the Completion of Part II Final Report on the Effectiveness of IDA Internal Controls for Assuring Reliable Financial Reporting, Compliance with IDA’s Charter and Policies, and Operating Efficiency and Effectiveness Volume II Completing Part II and Integrating Parts I and II 2009 The World Bank Washington, D.C. This paper is available upon request from IEG-World Bank. ©2009 The Independent Evaluation Group, The World Bank Group 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved This volume, except for the elements contributed by group and institutions outside IEG, is a product of the staff of the Independent Evaluation Group of the World Bank Group. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. This volume does not support any general inferences beyond the scope of this evaluation, including any references about the World Bank Group’s past, current, or prospective overall performance. The World Bank Group does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank Group concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The Independent Evaluation Group encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request to the Independent Evaluation Group. ISBN: 978-60244-111-8 Independent Evaluation Group Knowledge Programs and Evaluation Capacity Development (IEGKE) E-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on recycled paper Acronyms and Abbreviations AA Analytical and advisory activities GAC Governance and Anti-Corruption ARPP Annual Review of Portfolio Performance GAO U.S. Government Accountability Office BP Bank Procedure IAD Internal Audit Department BPM Business Process Module ICFR Internal Controls over Financial Reporting CAS Country Assistance Strategy IDA International Development Association CFAA Country Financial Accountability Assessment IEG Independent Evaluation Group CGAC Country Governance and Anti-Corruption IL Investment lending CODE Committee on Development Effectiveness INT Department of Institutional Integrity COSO Committee of Sponsoring Organizations IRMF Integrated Risk Management Framework (established by the Treadway Commission) ISR Implementation Status (and Results) Report CPAR Country Procurement Assessment Report IT Information technology CSR Controller, Strategy, and Resource KPI Key Performance Indicator Management LEG Legal Department DIR Detailed Implementation Review N/A Not applicable DPL Development Policy Loan OP Operational Policy ECDM Enterprise Content and Document OPCS Operations Policy and Country Services Management PR Procurement processes ELCQ Entity-Level Controls Questionnaire PRIMA Portfolio and Risk Management System EPR Evaluated pass rate RAPMAN Risk and Portfolio Management System ESW Economic and sector work QAG Quality Assurance Group F&C Fraud and corruption QSA Quality of Supervision Assessment FM Financial management SPC Strategy and Performance Contract FR Fiduciary Review SPR Simple pass rate TTL Task Team Leader Contents Evaluation Managers  Vinod Thomas Director-General, Volume II: Completing Part II and Integrating Evaluation Parts I and II  Cheryl Gray Director, Independent Evaluation Group-World About this Volume ................................................................................................... vii Bank Key Technical Terms ................................................................................................ ix  Nils Fostvedt Task Manager Section I: Entity-Level Controls Review (Concluding Part II) ............................... 1 Preamble to Section I ......................................................................................... 3 Annex A: Analysis and Evaluation of Management’s Approach and Method in Part II .......................................................................................... 4 Introduction .................................................................................................... 4 Mapping the Bank Units Comprising the Entity-Level Controls ..................... 4 The Entity-Level Controls Questionnaire ....................................................... 5 Other (Non-ELCQ) Evidence Used in Evaluating Entity-Level Controls ........ 6 Evaluating Management’s Approach and Method ....................................... 10 Annex B: Analysis and Evaluation of the ELCQ Results ............................. 13 Introduction .................................................................................................. 13 Statistical Analysis of ELCQ Responses ..................................................... 13 Evaluating Entity-Level Controls Using the Evaluated Pass Rate ............... 15 Analyzing the Negative Responses ............................................................. 16 Identifying Deficiencies from the ELCQ Responses .................................... 20 Section II: The Integrated Internal Controls Framework (Combining Parts I and II) ....................................................................................... 23 Preamble to Section II ............................................................................... 25 Annex C: Integrating parts I and II: Scope Limitations and Control Deficiencies ....................................................................................................... 27 Scope Limitation Issues Addressed ........................................................ 27 Additional Control Testing Completed .................................................. 27 AAA/ESW ............................................................................................. 28 Debt Sustainability Analysis ................................................................. 28 IDA Credit/Grant Mix ............................................................................ 28 Safeguard Risk Oversight..................................................................... 28 Other Transactions Testing Completed ................................................ 29 Compliance Issues Postponed to Part II ................................................. 29 Information Technology Controls .......................................................... 29 Field Office Controls ............................................................................. 29 Fraud and Corruption ........................................................................... 30 Internal Control Weaknesses ................................................................... 30 iii CONTENTS Overall Status ....................................................................................... 31 Material Weakness ............................................................................... 33 Significant Deficiencies ......................................................................... 33 Deficiencies .......................................................................................... 33 Part I Significant Deficiencies and Deficiencies .................................... 36 Part I Effectiveness and Efficiency Controls Issues .............................. 36 Annex D: Factors Combining to Form a Material Weakness ....................... 38 Introduction ................................................................................................ 38 I. The Bank’s Fraud and Corruption Initiatives: The Need for New and Innovative Approaches to Control ........................................................... 39 II. Deficiencies in Existing Controls ......................................................... 42 III. A Material Weakness in Controls over F&C ....................................... 45 IV. Summary and Suggested Remedies .................................................. 48 Annex E: IEG Composite Evaluation of the Internal Controls Framework .. 51 Description of Approach ........................................................................... 51 Rating the Effectiveness of IDA’s Controls ............................................. 51 Using the IEG Template to Rate the Controls Framework ..................... 53 Application of the Entity-Level Template: Summary of Results ............ 54 Annex F: Statistical Appendix ......................................................................... 59 Boxes Box A.1. Bank Units that Management Mapped to Structural Components of Entity-Level Controls .............................................................................. 5 Box A.2. Summary of IEG’s Evaluation of Management’s Approach and Method in Its Entity-Level Assessment .............................................................. 11 Box B.1. The Evaluated Pass Rate ..................................................................... 15 Box B.2. Initial List of Deficiencies Identified by IEG from Managements Questionnaire ....................................................................................... 21 Box C.1. Part I Findings Subject to Review and Modification in Light of Part II Findings ................................................................................................ 31 Box D.1. Standards and Evidence which Led to a Finding of a Material Weakness ............................................................................................. 47 Box E.1. The Four-Part Rating System ............................................................... 54 Box E.2. Questions Which Led to IEG Ratings of Moderately Satisfactory or Less on COSO Components ............................................................ 57 Tables Table B.1. Distribution of “No” Responses by Frequency ................................... 19 iv CONTENTS Table C.1. Weaknesses Identified in Twenty-One IAD Country-Level Audit Reports Issued July 2006 through December 2007 ............................. 30 Table C.2. Summary of Material Weaknesses and Significant Deficiencies Reported by Management, IAD, and IEG ............................................. 32 Table E.1. Ratings from the Template IEG Ratings of Management’s Assessment .......................................................................................... 54 Table E.2. IEG Rating of Overall Effectiveness of IDA’s Controls Framework.... 55 Table E.3. Rating Management’s Assessment of Controls over Efficiency and Effectiveness of IDA’s Operations ........................................................ 56 Table SA.1. Classification of Management's Questionnaire Responses (Based on Evaluated Pass Rates) .................................................................... 59 Table SA.2. The Distribution of "No" Responses by Frequency and COSO Components ......................................................................................... 60 Table SA.3. Percentage Distribution of Questions by Category of Responses* . 61 Table SA.4. Distribution of “No” Responses by COSO Components.................. 62 Table SA.5. Negative Responses by Type of Controls Issue: Control Design or Control Operation ................................................................................. 63 Table SA.6. Portfolio Efficiency Indicators .......................................................... 64 Table SA.7. Distribution of Entity Level Template Ratings.................................. 65 Figures Figure B.1. Simple Pass Rates by COSO Component ....................................... 14 Figure B.2. Effectiveness of Controls Based on Evaluated Pass Rates: Management versus IEG Interpretations .............................................. 16 Figure B.3. Negative Response Rates by Organizational Units.......................... 18 Figure C.1. Status of Deficiencies as of September 2008 .................................. 31 Figure D.1. Remedies to Combat a Material Weakness in Controls over Fraud and Corruption ...................................................................................... 49 Figure E.1. IEG’s Evaluation of the Effectiveness of IDA’s Internal Controls Framework ........................................................................................... 52 Figure E.2. Distribution of Template Ratings ...................................................... 59 v About this Volume Volume I presents the synthesis of IEG’s overall evaluation. By contrast, Volume II gives an account of the evidence, methodology, and underlying analysis that went into arriving at the overall findings shown in Volume I. Given that management decided to conduct its assessment in two parts (Parts I and II) IEG had to make an evaluation of each part separately, and then had to evaluate both parts as an integrated whole. IEG has already written and issued its two reports evaluating Part I. The present report concerns itself with the evaluation both of Part II and then of the integrated whole. Volume II has therefore been designed into two sections to fulfill those two key purposes, evaluating Part II and then the integrated whole: Section I provides a record of how the review of the entity level controls was conducted by management (comprising Part II of the overall review) and how IEG evaluated that assessment to arrive at its principal findings, based largely on management’s questionnaire results. The section contains two annexes, dealing separately with method and findings: Annex A: This deals with the type of approach and method that management used in its assessment of entity level controls. IEG conducted a specific evaluation of the approach and method to be sure that the methods were robust and did not lead to biases in findings and conclusions. Annex B: This annex deals with the evidence and findings of management’s entity level assessment, based mainly on the evidence of management’s primary instrument for conducting the entity level controls assessment (i.e. the Entity Level Controls Questionnaire (ELCQ)). The ELCQ was a questionnaire organized around the five COSO components, to be filled out by unit managers across the Bank. From its results, IEG conducted a statistical analysis to establish certain “pass rates” to show how effectively the entity level controls were perceived to be operating. The annex also describes how these results were used to identify certain deficiencies in entity level controls. Section II provides the basis for consolidating and integrating the results and findings from Part I and Part II, and contains IEG’s composite evaluation of the overall integrated controls framework. Section II contains four annexes: Annex C: This annex begins the integration of Part I and Part II and focuses mainly on controls weaknesses. It deals with certain scope limitations that applied to management’s approach to Part I (which would be weaknesses had they not been vii ABOUT THIS VOLUME addressed in Part II). It describes compliance issues that were postponed in Part I for completion during Part II. And it lists all controls weaknesses identified in both parts of the review: one material weakness, five significant deficiencies, and some 160 deficiencies found overall, and describes how many have been resolved and those whose remedies are still in progress or still to be acted upon. Annex D: This annex provides a detailed account of the background, evidence, criteria, and judgments relating to the IEG finding that there is a material weakness in the complex of controls that govern IDA’s efforts to ensure against F&C in its lending operations. Annex E: This annex contains IEG’s final composite evaluation of the integrated controls framework as a whole. It is based on all sources of evidence. This includes taking account of the underlying analysis presented in Annexes A and B (showing the ELCQ results), it integrates findings from both parts of the review, it considers other evidence available and it reflects the results of the Template application that IEG used to rate the effectiveness of the controls system. This is where, based on all sources of evidence, IEG gives its own independent evaluation of the effectiveness of the controls framework, as is summarized in Volume I. Annex F: This contains the Statistical Appendix with data gathered from the entity level controls review, mainly from the ELCQ. viii Key Technical Terms Audit Standards Criteria established by recognized accounting and audit bodies (in this case COSO and Accounting Standards 2 [AS2]) for conducting audits and reviews of internal controls that offer a basis for providing assurance that controls are well designed and working as intended, and for identifying deficiencies, significant deficiencies, and material weaknesses. Bottom-up Approach The approach adopted by management in its assessment did not begin with a top-down, entity-level review, but focused first on business processes at the transactions or operating level. Hence, it has been described as a bottom-up approach. Business Process Modules Management chose to conduct this review of internal controls by (BPMs) identifying the main business processes in which IDA is engaged on a daily basis in the course of its operations. There were 35 procedures in all, covering IDA allocation; the Country Assistance Strategy (CAS) process; the main lending products (Specific Investment Loans, or SILs, and Development Policy Loans, or DPLs); and the fiduciary, contractual, safeguards, and quality assurance processes that support lending. Each process was mapped and described as separate business process modules, each containing the key internal controls that are the subject of the review. Business Process Template A standardized assessment questionnaire and rating system used by IEG to provide quality ratings of management’s method and approach in identifying, describing, and mapping the business processes, and of its method in assessing the effectiveness of control design and of control operation. COSO Integrated Framework A framework of management principles (COSO components) in an (“Internal Controls-Integrated organization that, when collectively operating as intended, will Framework”) provide reasonable assurance as to the attainment of three key organizational goals (COSO objectives): reliable financial reporting, operational effectiveness and efficiency, and compliance with laws and regulations (in IDA’s case, with its charter and internal policies and procedures). The COSO components are: Control Environment, Risk Assessment, Control Activities, Monitoring and Learning, Information and Communications.1 Deficiencies, Significant Design flaws, omissions, or noncompliant operation of controls, Deficiencies, Material discovered in the course of a controls review, denoting an ascending Weaknesses order of seriousness. The precise criteria by which the three categories of materiality are distinguished are explained in Annex B of the Part IA Report. However, in the case of operational as against ix KEY TECHNICAL TERMS financial reporting, there are no such clear yardsticks by which to measure the materiality of a given weakness or set of weaknesses. Some judgment is required. The criteria to be used as a guide in making the needed judgments are those outlined in Annex B of the Part IA report. Entity-Level Controls Entity level controls refer to those internal controls applicable to the entity as a whole (i.e., ”high level” controls). As such, appropriate entity level controls established and supported by management are a critical ingredient in creating an effective control environment. Examples of entity level controls include creating effective systems and processes for performance management (performance measurement and results), human resource management (hiring, performance evaluation, and training), and ethics (code of conduct and ethics regulation). Examples also include the creation of control units with responsibilities that cut across the organization and exist for the purpose of monitoring the effective achievement of objectives and/or implementation of internal controls such as IEG, IAD, QAG, INT and others. Entity-Level Controls A questionnaire designed by management to be answered by Questionnaire (ELCQ) managers throughout the operating units in the Bank, with questions aimed at soliciting opinions from managers about the effectiveness of controls. Where questions received “yes” responses the presumption is that the control in question was seen to be working, and where “no” or qualified responses were given, there was presumed to be a weakness in the control. Entity-Level Template A standardized questionnaire and rating system used by IEG to evaluate and give quality ratings to both management’s approach and method in its assessment of the entity-level controls framework, and to evaluate the strengths and weaknesses of the framework, as viewed across the five COSO components. Evaluated Pass Rate (EPR) An ELCQ question about a given control could be answered “yes” by some managers while being answered “no” by some others. The EPR is the number of questions deemed to have been answered “yes” on balance, taking into account also the number, type and reasons for the “no” responses given for the same question, as a percentage of the total number of questions. Since IEG and management used different criteria for making these judgments, the EPRs calculated by each party were different. Evaluation Panels In applying its Business Process Template, IEG assembled panels of 3-4 people, including controls specialists, and with experts in the particular discipline covered by the given BPM. The panels arrived at consensus judgments on the ratings that should be applied to each section of the module, according to their evaluation of the materials presented by management. Exceptions Non-compliances deemed to be of a less serious or material nature than deficiencies. Exceptions/Deficiency Rates The number of exceptions/deficiencies found during the Part IB testing of key controls, divided by the number of control steps in the x KEY TECHNICAL TERMS sample. Internal Controls Controls, individually or collectively, are structured means within an organization to enable it to achieve its business objectives while addressing risk. Control instruments include the control framework (in IDA’s case, the COSO framework), organizational checks and balances, published policies, and required procedures, among others. Integrated Internal Controls The combined system of key controls contained in the transactions- Framework level business processes and the entity level controls that provide for governance of the organization as a whole Key Control A gateway and decision point, involving key units and IDA staff, in a given business process module, through which a business transaction being processed must pass. It is the effectiveness in design of these controls and the subsequent testing of the effectiveness of their operation that is at the center of this review. Non-compliances Controls or control steps found during testing to be not operating in conformity with the design of the control. The concept of non- compliance includes both exceptions and deficiencies. Process Map The flow chart that graphically depicts all steps in a business process module. Review The term used to refer to the entire process of this study. Management conducted an assessment, the Internal Audit Department (IAD) conducted a review and opinion, and IEG conducted an evaluation. When referring to all three processes as an entity, the term used is “review.” Risk Focal Points In the adaptation of the COSO framework by the Bank and IDA to meet their own needs, management has defined and added to the framework four key points of risk that face the mission of the Bank Group and are especially relevant to IDA. These are: Strategy Effectiveness, Operational Efficiency, Financial Soundness, and Stakeholder Support. Simple Pass Rates (SPR) The number of “yes” responses received for each category of responses in the Entity-Level Controls Questionnaire, divided by the total number of responses. The SPR therefore gives a direct measure of what the ELCQ results show regarding the effectiveness of controls; hence it directly reflects the perceptions of the managers responding to the questionnaire. Walkthrough An interactive interview and review of process documentation conducted by management with relevant teams of IDA staff knowledgeable in a particular business process and its associated controls, with a view to verifying that controls are designed in the way described and operate as intended. 1. See World Bank Web site for COSO Framework. xi SECTION I Entity-Level Controls Review (Concluding Part II) Preamble to Section I IEG recognizes that the Bank and IDA have clear missions and are generally well structured and equipped to pursue them. IDA’s general mission statement stems from its Articles and it has a set of policies and procedures to guide its pursuit of strategic objectives related to that mission. It also shares in the Bank’s organization and operational matrix management structure (Regions and Networks). As the world leader among development finance institutions it is presumed to have excellent, professional management and staff as well as a control environment that assumes integrity, ethical values, discipline, and human resource policies and practices to support the pursuit of its operations. IDA has explicit mechanisms and routine processes to assess various types of risk that must be faced in pursuing its goals. To match and manage these risks, IDA shares in the Bank’s policies and procedures and has a structured budget and resource allocation system. That system aims to ensure that resources are deployed with due regard for efficiency and in accordance with IDA’s strategic goals. IDA’s Board receives quarterly updates on those resource deployments. IDA has its own allocation model to deploy the resources that donors provide on the IDA replenishment cycle. In recent years, the Bank has strengthened its capability for monitoring various aspects of its activities, including with the creation of new units such as the Quality Assurance Group (QAG), Department of Institutional Integrity (INT), and Inspection Panel (IP) and new areas of focus, including those covered by IEG. Finally, as a major knowledge institution the Bank has modern information and communications systems connecting a worldwide network of field offices with headquarters and a wide range of audiences for its activities. The purpose of the review has been to document, verify, and test IDA’s controls mechanisms and processes, and to identify areas where controls may not always be working as intended, or where new controls may be needed. It is in this light, therefore, that IEG has viewed the output from management’s assessment. 3 Annex A: Analysis and Evaluation of Management’s Approach and Method in Part II Introduction 1. This annex deals with the type of approach and method that management used in its assessment of entity level controls. IEG conducted a specific evaluation of the approach and method to be sure that the methods were robust and did not lead to biases in findings and conclusions. 2. The main tools management used in its assessment were questionnaires and analysis of various reports, papers, and documents, specifically it:  Mapped all Bank operational and central control and monitoring units and linked them to the COSO components to define the entity-level controls framework;  Administered two questionnaires for the Bank’s operating, service and control units:  The annual Internal Controls over Financial Reporting (ICFR) questionnaire (routinely used for financial reporting), was used as a background reference;  A new Entity-Level Controls Questionnaire (ELCQ) was designed especially for the IDA review, focusing on operational (not financial) reporting issues at the entity level;  Interviewed unit staff to gather more detailed information about their questionnaire responses;  Prepared analytical papers on such special topics as the budget process, and the role of the Board;  Reviewed the results of analytical and evaluative work conducted over the years by central control units as part of their ongoing work programs, including investigative- oriented reports and Internal Audit Department (IAD) audit reports. Mapping the Bank Units Comprising the Entity-Level Controls 3. Management identified 31 Bank units (shown in Box A.1) whose roles and functions contribute to the entity-level controls framework. The Board of Executive Directors was not included as management did not intend to have the Board respond to the questionnaire, and the Board’s role was handled in a separate process. Management’s assessment of the entity- level controls examined the mandate and terms of reference of each of these units (or groups of units) as well as recent outputs from their ongoing work programs. The findings were used to assess their respective contributions to IDA meeting the COSO principles and to 4 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II establish where weaknesses might exist in the exercise of their internal controls function. Management also sent these 31 units the ELCQ. IEG agrees that these units embody the functions that constitute the entity-level controls system over IDA operations. Box A.1. Bank Units that Management Mapped to Structural Components of Entity-Level Controls Three Senior Six Regions Four Networks Nine Central Control Nine Other Units Management Units Units 2 Managing Directors Africa (AFR), East Asia Human Development Controller, Strategy Treasury (TRE), (MDD and MDW)*, and and the Pacific (EAP), (HDN), Finance and and Resource External Affairs (EXT), the Chief Financial Europe and Central Private Sector Management (CSR), Human Resources Officer Asia (ECA), Latin Development (FPD), Quality Assurance (HRS), Information America and the Poverty Reduction and Group (QAG), Services Group (ISG), Caribbean (LCR), Economic Independent General Services Middle East and North Management (PRM), Evaluation Group Department (GSD), Africa (MNA), and Social Development (IEG), Internal Auditing Development South Asia(SAR) (SDN) Department (IAD), Economics (DEC), Concessional Finance Legal (LEG), Corporate and Global Secretariat (SEC), Partnerships (CFP), World Bank Institute Ethics and Business (WBI) Conduct (EBC), Operations Policy and Country Services (OPC), Institutional Integrity Department (INT), Inspection Panel (IP) *Graeme Wheeler and Juan Jose Daboub. A third Managing Director, Ngozi Okonjo-Iweala, joined the Bank only in late 2007 and therefore was not included among the respondents. The Entity-Level Controls Questionnaire 4. Questionnaire Design: The ELCQ was used to deepen the evidence from management’s own Bank unit assessment based on the mapping review. IEG was asked to comment on the draft questionnaire during the design phase before it was distributed to the Bank units. Once the questionnaires were completed, IEG made a more comprehensive evaluation of its effectiveness, finding it to be appropriately organized—with questions arranged in groups under each COSO component—and with questions that were generally well focused on the key issues in each area. Through this organization, the questionnaire was able to deliver results that were easily aggregated into COSO topic areas. 5. The questionnaire contained 157 questions. Based on the population of the 31 Bank units, this would give a total of 4,867 potential responses. However, a number of questions did not apply to all units, so the effective universe was 4,149 potential responses. (This total included 79 questions that were asked to units, but which they decided were not applicable to them.) IEG notes that some of the questions that were not asked of all the monitoring units possibly could have been asked, particularly of INT, and might in that case have yielded useful information. 5 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II 6. There are also some issues with the questionnaire approach beyond the appropriateness of its design. First, questionnaires record perceptions of managers, not hard facts. Second, there can be issues of candor, and respondents may have shown some degree of bias, because they are often “owners” of the controls and processes being examined. Finally, even if candid, managers may miss certain realities because they may focus on their units more than on the institution as a whole (as in one important case with question 68 relating to the quality and currency of Operational Policies/Bank Procedures [OP/BPs]). 7. These issues are typical for most questionnaire approaches. In this case, management’s assessment did not rely solely on the evidence provided by the questionnaire. Also, such questionnaires are commonly used in controls reviews in the professional auditing industry. On balance, therefore, IEG is satisfied that the questionnaire approach was useful and revealed relevant and actionable information. Moreover, in IEG’s follow-up interviews with a number of the respondents it became evident that they had treated the questionnaire seriously, senior staff answered the questions after careful thought and discussion within each responding unit, and the process was itself a learning experience for some units. 8. In any future surveys of this kind IEG would suggest some refinements:  Be more explicit as to whether respondents should speak about their units, or about the institution, and what should be the basis of their responses, i.e. relevant experience and observations in their current areas of responsibility or also from previous positions or opinions of others.  Distinguish more clearly those questions that are searching for responses about controls design and those that are about the operation within the particular work unit of controls.  Eliminate overlaps between questions, and reformulate questions that were presented in the form of multiple questions when respondents might have given different responses had the questions been asked individually. Other (Non-ELCQ) Evidence Used in Evaluating Entity-Level Controls 9. Consultation with Bank units: The management team interviewed a number of senior managers who had completed the questionnaire in order to gain a more in-depth view of their responses and what lay behind them, particularly in areas where ELCQ responses suggested possible control weaknesses. These discussions also often went beyond the questionnaire itself to discuss aspects of internal controls more generally, which frequently added significant value to what the questionnaire responses had covered. 10. Analysis of control units’ reports: Among the Bank units that management had mapped to the organizational structure of entity-level controls, the central control units were considered pivotal, specifically in the entity-level monitoring and control functions. Management produced an internal paper that summarized recent work and findings conducted by these units (EBC, CSRRM [Resource Management], CSRSI [Corporate Strategy and Integrated Risk Management], CTR, OPCFM [Financial Management], QAG, IAD, INT, IEG, IPN). This work provided the basis to assess the effectiveness of these units in their 6 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II contribution to entity-level controls and in providing feedback, including recommendations for controls improvement, to the Board of Executive Directors and Senior Management. IEG Observations: Management’s summary document on the central control units gave ample evidence of the breadth and depth of information provided on an annual basis by these units, which was useful to corroborate and give context to the questionnaire response data. 11. Review of Board of Executive Directors’ role in the control framework: This paper, written jointly by Legal and the Secretary’s Departments (LEG/SECBO), described the roles and responsibilities of the Board of Executive Directors. The Board was not subjected to a review and examination and it was not asked to respond to the ELCQ. Management did, however, take due note of the major Board oversight functions and the Board contribution to the Control Environment of the COSO framework. 12. Analysis of the Bank’s budgeting processes: Management commissioned a background paper that described the regular annual budget processes. It highlighted their significance for translating strategic goals from the Medium-Term Financing Strategy (MTFS) into unit operations programs (through the Strategy and Performance Contract, or SPC, and monitored in the Quarterly Business Review, or QBR). It also highlighted the attention given to operational efficiency, in the form of budgetary allocations linked to previous performance, and including specific efficiency incentive measures such as the efficiency or “productivity” tax. IEG Observation: IEG commissioned its own working paper on this topic, which gave very similar insights. IEG also gathered specific efficiency indicators (see Table SA.6 in the Statistical Appendix), which it used as a reference point to show the type of information that Senior Management is provided on a routine basis. 13. Other evidence gathered: Management also used reports and recommendations stemming from parallel but unrelated studies, such as the Volcker Report on the role of INT, the Detailed Implementation Review (DIR) on the India Health Sector recently completed by INT, and some of the work being undertaken on fraud and corruption issues under the Governance and Anti-Corruption (GAC) initiative. IEG has also drawn on these sources. 14. In addition to reviewing summary documents and analytical papers prepared by management and conducting its own interviews, IEG also independently selected and reviewed certain central control unit reports that were directly relevant to the IDA controls review, namely:  IAD audits of country operations in field offices: IEG reviewed 12 of 21 IAD audit reports (from FY07 to Q1 FY08) relating to IDA-only countries which had been earlier reviewed by management. Of the 12, only one was rated satisfactory, 11 were rated needs improvement. Management’s review of the full 21 showed ratings of 5 satisfactory, 14 needs improvement and 1 was unsatisfactory (and one was not rated). (Of the country cases rated less than satisfactory the dominant area in which deficiencies were identified was in project supervision, and specifically in the area of financial management (FM) and procurement (PR). In one case, administration of the field office budget was unsatisfactory.) IEG Observation: This raises an issue regarding controls over supervision of FM and PR, and corroborates the Part I finding that the fiduciary Business Process Modules (BPMs) were among those with the highest rates of noncompliance. 7 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II  QAG Reports on Project Supervision (QSA 7): QAG reported that supervision had improved—the share of projects with unsatisfactory supervision (moderately unsatisfactory, unsatisfactory, and highly unsatisfactory) was only 5 percent as of 2007. However, only 50 percent of the projects were rated satisfactory or higher; and QAG reported that Implementation Status Reports (ISRs) understated the riskiness of the portfolio and that more projects needed special attention and possibly more resources. The QAG report pointed out a need to focus on development effectiveness outcomes; a need for improved quality of data to support clear key performance indicators (KPIs) and enable teams to address threats to achievement of development objectives (DOs); a need for improved sector management oversight; a need for quality supervisions skills, particularly better support to local staff in fiduciary, social, and environment matters; a need to expand budgets for supervision; and a need for more candor in ISRs. IEG Observation: QSA 7 reflects improvements in supervision quality over some past years but in some key areas (candor, need for more budget resources) weaknesses may reinforce controls deficiencies found in other areas (lack of controls or lack of observance of controls over fraud and corruption in particular).  INT Detailed Implementation Reviews: IEG reviewed all DIRs/FRs completed by INT, including the recent DIR on the health sector in India. 1 These reports provide detailed examinations of the implementation of Bank and IDA projects, including through forensic audits. The results shed light on the susceptibility of IDA projects to fraudulent activity and raise issues related to the supervision of projects, since, as in the case of the India DIR they have identified indicators of fraud and corruption in all six sets of projects studied. From the perspective of the IDA controls review, the questions raised by these studies are: (a) which Bank controls (policies, procedures) were breached such that fraud and corruption could have occurred and not been detected in a timely manner? and (b) How widespread might such practices be in other sectors and countries? IEG Observation: The evidence emerging from DIRs/FRs has contributed to conclusions reached in this review that controls in place have not reasonably mitigated fraud and corruption risk at several levels of the Bank’s operations processes, which IEG has judged to be a material weakness in the control system. 15. Background papers commissioned by IEG: In addition to the several papers presented as relevant evidence for the entity-level controls review, IEG also commissioned four background papers on the following topics:  Budgetary Process and Efficiency Measurements: The paper describes the Bank’s mechanism for setting strategic goals, translating these into VPU and working-level goals, allocating resources through the annual MTSF paper, and the monitoring of progress on goals through the associated SPC and QBRs. It also describes the specific efficiency goals and incentive mechanisms, including the shift to dollar budgeting and the efficiency tax recently introduced. IEG Observation: The Bank has an articulated budget mechanism and a regular high-level review cycle that monitors quarterly progress, and the budget contains efficiency incentives.  Development Effectiveness of IDA: The paper draws heavily on IEG reports on annual reviews of development effectiveness (ARDEs) for the past several years, and tracks 8 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II achievements in IDA effectiveness since 1990. The paper offers a clear account of IEG’s role as one of the Bank’s chief entity-level units concerned with measuring and tracking development effectiveness, and it contains selected evidence of actual outcomes, which is relevant to the overall extent to which the controls over IDA operations can be credited with being effective. IEG Observation: The Bank has a well-defined mechanism through IEG for tracking operational effectiveness. This is reported on an annual basis in a way that keeps management and other stakeholders well informed.  Review of Operations Evaluation (AROEs): The paper reviews IEG’s Annual Report on Operations Evaluation (AROE) from 1997 through 2006 to provide an overview of the evolution of the Bank’s systems to manage for development effectiveness and on IEG’s effectiveness over this period. The AROEs covered several themes, including lending, analytical and advisory activities (AAA) and economic and sector work (ESW), country focus and the Country Assistance Strategy (CAS), and sector strategies, trust funds and partnerships, and corporate performance evaluation. The AROEs also contain self-evaluations by IEG of its own effectiveness, and the background paper presents survey data on the perceptions of the operating staff as to the timeliness and quality of IEG products. IEG Observation: The AROEs have provided a regular and transparent account of the Bank’s evaluation effectiveness. IEG has now merged the separate AROE into the ARDE discussing issues concerning Bank operations evaluation and development effectiveness in a single annual report. The key analytical and reporting aspects of the ARDEs will be maintained but in a more streamlined reporting format.  Results-Based Management: The paper describes the Bank’s significant progress on the adoption of a results agenda and the mainstreaming of the Monterey, Marrakesh, and Paris initiatives. Most CASs have now adopted a results-based approach (RBCAS), and the Results Measurement System (RMS) is being used by IDA, which is somewhat ahead of the IBRD in this respect. There is, however, more to be done, as was also confirmed by the questionnaire. Areas where improvements are needed include:  Extending key performance indicators (KPIs) to the individual level;  Extending the Results Monitoring Learning System Bank-wide;  Improvement of the results links from projects to sectors to country outcomes;  More systematic definition of project development objectives in AAA and tracking of impacts;  A shift in managerial emphasis from the current high level of attention to performance measurement (data generation) toward more focus on performance management (i.e. usage, effective implementation). IEG Observation: The Bank has made a good start in introducing results measurement systems and is engaged, alongside other international financial institutions, in implementing a results-based agenda. However, the instruments have yet to be fully developed at the operations level. 9 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II Evaluating Management’s Approach and Method 16. In summary, management conducted its assessment of entity-level controls by identifying, mapping, and surveying the work output from the units in the Bank that contribute to the entity-level controls framework. It used two questionnaires (ICFR and the ELCQ), but principally the latter, which was designed for the purpose, to corroborate and deepen its findings. Using this two-pronged approach, management assembled a significant body of information, consisting of a compendium of reports on the work outputs of the central control units in the Bank, supplemented by a number of background and other papers, including the Volcker Report on INT, IAD audit reports, and reports from QAG and IEG. 17. IEG’s Evaluation: IEG used its Entity-Level Template to rate the quality of management’s approach to assessment. The question IEG posed was: Were there any aspects in management’s approach and method in assessing the entity-level controls framework that would give rise to significant doubts as to the validity of the results of that assessment? IEG concluded that this was not the case, and it rated the method overall fully satisfactory, but with some suggestions for future improvements in the questionnaire design. Overall, management’s approach was sound and its methods and tools were useful and responsive, and IEG finds them not to be the cause of significant doubt as to the results obtained. Questionnaires of this kind are commonly used in the auditing profession as tools to examine entity-level controls. 18. Template ratings: Ratings were applied to two separate aspects of management’s approach, with the results as shown in Box A.2. The observations and suggestions made to improve the questionnaire referred to those few areas of possible ambiguity in the questionnaire, as discussed in paras. 4-7. The rating for both the Design of Assessment Instruments and for the Use of Data from the Questionnaire and Other Sources was fully satisfactory. 10 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II Box A.2. Summary of IEG’s Evaluation of Management’s Approach and Method in Its Entity-Level Assessment Approach/Instrument Evaluated by IEG IEG Evaluation Questionnaire Design:  Despite some shortcomings—cases of  Blank form draft: commented on scope, overlap, unit/institutional focus, some content, responsiveness to COSO multiple questions—the questionnaire is components and linkages to transactions found to be a useful instrument for this level review.  Integration of Part II questionnaire with ICFR questionnaire  Suggested inversion of one question Questionnaire Response Matrix:  Matrix found to be comprehensive,  Raised issue of candor in questionnaire consistent, and transparent tool to present responses, and consistency between unit and aggregated response data. Grouping of units Bank perspectives could have been aligned more closely to  Reviewed the matrix format summarizing COSO. responses  Confirmed that the universe of Bank units responding was appropriate  Clarified consistency of matrix summary and questionnaire responses Interviews:  Management interviews provided clarifications of answers and not  IEG did not attend management interview corroboration and verification that controls sessions were effective, but use of other data sources  Conducted its own selected interviews allowed supportable conclusions to be drawn. Overall Method: IEG Ratings of Management Approach  Applied IEG Entity-Level Template to rate Overall rating Design of Use of data different aspects of management’s approach assessment in assessing each COSO component instruments  IEG used these ratings to confirm its own 1 1 1 evaluation of the quality and strength of the Rating Scale entity-level controls framework 1 = Fully satisfactory  Key Qualifications: Few details on links 2 = Satisfactory with qualifications between entity and transactions levels; 3 = Moderately satisfactory treatment of efficiency and effectiveness 4 = Unsatisfactory made no reference to outcomes CONCLUSION IEG concludes that management’s method and approach provides a credible, transparent, and robust basis for the assessment and evaluation of entity-level controls. IEG rates management’s overall approach fully satisfactory, but with some observations relating to improvements that could be made in questionnaire design. 11 ANNEX A ANALYSIS AND EVALUATION OF MANAGEMENT’S APPROACH AND METHOD IN PART II Annex A 1. These DIRs/FRs were not randomly generated, but were initiated where F&C problems were perceived to exist. Two cases were initiated by INT, the remaining four were initiated at the request of the regions. 12 Annex B: Analysis and Evaluation of the ELCQ Results Introduction 1. This annex deals with the evidence and findings of management’s entity level assessment, based mainly on the evidence of management’s primary instrument for conducting the entity level controls assessment (i.e. the Entity Level Controls Questionnaire (ELCQ)). The ELCQ was a questionnaire organized around the five COSO components, to be filled out by unit managers across the Bank. From its results, IEG conducted a statistical analysis to establish certain “pass rates” to show how effectively the entity level controls were perceived to be operating. This annex also describes how these results were used to identify certain deficiencies in entity level controls. 2. One advantage of the questionnaire method was that it provided an extensive database of responses, reflecting managers’ perceptions across the full spread of relevant Bank units. IEG conducted an extensive statistical analysis of the data. It examined the aggregate positive responses as a means of establishing the extent to which controls were seen to be working well. It also looked at the incidence, nature, and frequency of negative responses to identify areas where control weaknesses appeared to exist. From this analysis, IEG was able to compile an initial list of deficiencies in the entity-level controls. In corroboration with evidence from other sources, and after integrating all the findings from Part I and Part II, it formed the basis for IEG’s evaluation of the effectiveness of the overall integrated internal controls framework, which is the subject of Section II and Annex C. 3. The present annex summarizes the statistical analysis of the ELCQ results (both positive and negative responses) and summarizes, by COSO components, the negative responses and how these have been used to identify controls deficiencies. Statistical Analysis of ELCQ Responses 4. Given the nature of the questionnaire approach, and the potential (even if minor) for some degree of bias in the results, IEG used three approaches to analyzing the results of the ELCQ: it started by using a simple pass rate, then moved to establishing an evaluated pass rate, (both of these based on ELCQ results only) and then it undertook a full rating and composite evaluation of results using all forms of evidence, including the ELCQ, but also using evidence from other sources. This third step is dealt with in detail in Annex E. 5. Simple Pass Rate: The simple pass rate measures the aggregate number of “yes” responses received from all questions in the ELCQ, as a percentage of the total number of 13 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS individual responses received. What this measures, therefore, is the frequency of positive responses out of total responses, and gives a very general reflection of the perception of managers on the overall effectiveness of the controls system. The results from the ELCQ show that there were 3,673 positive responses in aggregate out of a total of 3,850 responses. This calculates to a simple pass rate of 95 percent and indicates that managers perceive that entity-level controls operate with a high degree of effectiveness. This pass rate compares to a simple pass rate of 93 percent for the compliance of key controls in the business processes at the transactions level. Since management organized the ELCQ around the five COSO components, the question was also addressed of whether effectiveness of controls might be different across the five COSO components. The SPRs were therefore calculated and shown in Figure B.1 below. The differences were not marked (from 92 percent for Control Environment to 99 percent for Monitoring and Learning) and all results were above the 90th percentile. Figure B.1. Simple Pass Rates by COSO Component 100 99.6 98 97 96 95.6 94 93.8 92 92 0 90 Control Risk assessment Control activities Monitoring Information and environment communications Source: IEG calculations based on Table SA.1 in the Statistical Appendix. 6. Evaluated Pass Rate: The ELCQ questions were each designed to shed light on whether managers saw a particular control as working effectively (in which case they would answer “yes” to the question), or whether they saw weaknesses or other issues (in which case they would answer “no” or with some qualification. Many questions received both “yes” and “no” or other responses, because managers differed in their view of the controls, depending on their respective perspectives. Therefore, in order to decide whether a given question had been answered overall and on balance positively or negatively, a judgment had to be made. This judgment was based on the number of “yes” and “no” responses, the origin of the “no” responses in terms of which units saw problems (and how close they were to the given control in question), and the evidence and reasoning given for 14 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS the negative responses. The pass rate that was calculated using this approach was therefore referred to as an evaluated pass rate (EPR). 7. The ELCQ contained 157 questions in all. Consistent with a simple pass rate of 95 percent, most of these (87 out of the total of 157 questions) received no negative responses at all. However, where a question received some negatives or “no” responses, the question becomes when to make the judgment that that question has, on balance, overall been answered “no” because of either the number of “no’s” or the importance of the “no” response either in terms of who gave the response, and/or the nature and materiality of the reasons behind the response. It was a judgment call to make these determinations, and management and IEG used somewhat different criteria in making their respective judgments. As shown in Box B.1, management concluded that fully 150 out of 157 questions had received “yes” responses (an EPR of 96 percent), while IEG concluded that 144 questions had been positive, an EPR of 92 percent. Box B.1. The Evaluated Pass Rate Concepts underlying management and IEG assignment of results Based on questions and other data: Total number of questions (out of 157 questions) deemed to have received positive responses overall. Judgments were based on the number of units responding “no” (or ambiguously) to a given question as well as the nature and materiality of the responses (as contained in the ELCQ narrative). Management and IEG used somewhat different criteria in making their judgments and came to different conclusions regarding the categorization of questions. Management had a two-part classification (“yes” or “no”), while IEG had a four-part classification: Either “yes” or “no” or “Yes, but” (responses were positive but with significant qualifications) or “No, but” (responses were negative responses and/or negative comments judged to override the overall ’yes’ rating by management. Overall rating Yes Yes, but No, but No Total Management 150 0 0 7 157 IEG 103 41 9 4 157 Source: IEG calculations based on Table SA.1 in the Statistical Appendix Evaluating Entity-Level Controls Using the Evaluated Pass Rate 8. The outcomes shown in Box B.1 reflect how management and IEG would respectively rate the effectiveness of the Bank’s entity-level controls if only ELCQ data were used in the form of an evaluated pass rate, without looking further at the negative responses and without consulting other evidence. IEG used these results by taking the classifications shown in the Box and transforming them into the same four-part rating system IEG used in its Templates. (“Yes” was rated “1”; “No” was rated “4”; and the two intermediate responses were “2” and “3” respectively.) This was done for the different categories across all COSO components and the results were then tabulated into a statistical “diamond” to 15 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS depict how each component would be rated compared to the others. The results are shown in Figure B.2. It shows that IEG has consistently rated the effectiveness of controls in each component less positively than has management, though not by a significant margin. Management’s approach to the EPR resulted in an overall IEG rating of “1.1” (almost fully satisfactory); IEG’s approach gave a rating of “1.5” which was midway between fully satisfactory and satisfactory with qualifications. Figure B.2. Effectiveness of Controls Based on Evaluated Pass Rates: Management versus IEG Interpretations Control Environment 1. 2. Information and Risk Communication 3. Assessment 4. Management IEG Overall Ratings: Management: 1.1 IEG: 1.5 Monitoring Control and Learning Activities Rating Scale 1 = Fully Satisfactory 2 = Satisfactory with Qualifications 3 = Moderately Satisfactory 4 = Unsatisfactory 9. These ratings are of interest only as an interim indicator. The entity-level framework cannot be fully evaluated without a closer examination of the specific controls weaknesses that have been identified. A first pointer as to where these weaknesses occurred was given by the number of negative responses in the ELCQ for each component of the COSO framework. The following section describes the analysis of these negative responses. Analyzing the Negative Responses 10. Clearly, all responses to the ELCQ questions are important. However, since the controls review has the purpose to assess effectiveness of controls and to identify weaknesses, priority has been afforded to the negative responses—i.e. the residual 5-8 percent out of the 92-95 percent pass rates—because they signal possible weak spots in the 16 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS framework. Further, in making the judgments that underlay the EPRs, it was useful to view the incidence and nature of the “no” responses, to better gauge their importance in given cases, by asking the following questions:  Which Bank units gave “no” responses? One interesting perspective is to know in which Bank units these “no” responses originated. Did they come mainly from operational units or central, entity-level control units?  Types of controls issues: IEG also examined what type of controls issues were being addressed—whether issues of control design or control operation—a factor that will affect what type of remedy may be sought in the case of controls weakness.  Were “no” responses widespread among questions or concentrated in a few? This may reveal whether many controls had some minor weaknesses (one or two managers saw weaknesses) in some areas, or rather a few controls were widely seen not to be working adequately.  Identifying potential deficiencies: Not every “no” response can be taken to show that a deficiency exists. IEG has sifted through all “no” responses and compiled a list where it thinks deficiencies do exist, based also on the concentration of “no” responses.  Implications for the COSO framework: Are controls in all COSO components present within the framework with equal effect, or are controls in some components weaker than others? 11. Which Bank units gave “no” responses? This question is important because not all units will have the same perspective on all the issues raised by questions in the questionnaire. IEG posed the following hypothesis: The central control units, senior management, and the other service units have an institutional perspective as part of their function and are more likely than are the operating units to see fault with controls. 12. The data in the charts below clearly uphold this hypothesis. Figure B.3 A, which gives the absolute number of “no” responses, shows that the control units had 66 out of a total of 177 negative responses, but when combined with the other central units the total was 132, or 75 percent of the total. The hypothesis is equally upheld when looking at the negative response rates, which eliminates any bias in the data resulting from the fact that the unit groups are of different sizes, so larger groups may be expected to have a larger absolute number of negative responses. Figure B.3 B shows that both senior management and the central control units have negative response rates substantially higher than the average. 13. The data from these charts is useful in pointing to a related issue regarding the questionnaire method—whether this would have introduced a bias in the results because responding managers may have been less than candid in giving their responses. As shown in the in charts, the operating units together had a negative response rate below the average. However, managers in these units did comprise some 25 percent of all negative responses, which is not a trivial number and which suggests that a lack of candor may not have been a significant issue overall, a finding that was also corroborated by the interviews that IEG conducted with 14 responding units within the Bank. 17 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS Figure B.3. Negative Response Rates by Organizational Units A. (9 units) 70 66 60 50 Number of "no" responses (9 units) 40 (6 units) (3 units) 35 30 31 30 (4 units) 20 15 10 0 Networks Regions S enior Other Units Control Units Management B. 8.0% 7.5% 7.0% 7.0% Percnet of total effective responses 6.0% 5.0% 4.0% 3.6% 3.1% 3.0% 2.7% 2.0% 1.0% 0.0% Networks Regions S enior Other Units Control Units Management Source: Management Results Matrix. In B, the numerator is the number of “no” responses in each group, and the denominator is the total number of effective questions asked each group (i.e., excluding those not applicable to some units). 18 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS 14. Type of control issue: control design or control operation? When respondents to the questionnaire give a negative response, signifying that they believe a control is not operating effectively, it makes a difference to know whether it is the design of the control or the operation of the control that is being questioned. In the first case, the necessary remedy would be to put in place a new or improved control design; in the second case, the necessary remedy would be more effective enforcement of the existing control, for example, by heightened management supervision of a given process. Some direct testing of entity-level controls to determine the effectiveness of both their design and operation could have improved the results of the review and the basis for conclusions reached. Knowing whether the questionnaire is seeking perceptions about the design or operation of the controls could be helpful when developing plans for testing controls in the future, both at the entity and transaction levels. 15. IEG analyzed the negative responses and found that most (about 60 percent) were questioning the design—or the absence—of the control rather than its operation. The pattern was also similarly found across all COSO components. (See also Table SA.5) in the Statistical Appendix.) 16. Are “no” responses widespread or concentrated in a few areas? If negative responses were widely dispersed across all questions, but with only one or two “no” responses per question, it might be difficult to conclude that deficiencies existed. By contrast, if some questions had many “no” responses, this would suggest that there was a common view that a weakness existed in a given control. 17. IEG has analyzed the dispersion of the “no” responses across all questions. It found, as shown in Table B.1, that out of 157 questions, only 70 questions had any negative responses; 87 questions had nothing but positive responses. Of the 70 questions with negative responses, 52 had only one or two “no” responses, while 10 questions had between three and five “no” responses and 8 questions had more than five “no” responses. The table also shows that no less than 60 percent of all negative responses were found in the areas touched on by these 18 questions. In the section that follows, IEG uses this finding as a guide in pointing to where the more serious controls weaknesses may have been identified. Table B.1. Distribution of “No” Responses by Frequency Number of “no” responses/question TOTAL 1-2 3-5 >5 Total number of questions asked 157 Total number of questions with at least one “no” response 70 52 10 8 % distribution by frequency 100 77 13 10 Total number of “no” responses 177 69 42 66 % distribution 100 40 22 38 Source: IEG calculations based on management results data. 18. IEG’s statistical analysis of the ELCQ results has provided useful insights into the overall perceptions of managers in the Bank regarding the effectiveness of entity-level controls. It has also offered confirmation that the identified weaknesses in the questionnaire 19 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS approach have not caused any significant distortions in the results. Further, the collection of “no” responses which are contained in the ELCQ results matrix has provided a basis to identify specific control weaknesses and to judge their materiality. Identifying Deficiencies from the ELCQ Responses 19. This last section of Annex B contains a detailed record of how IEG used the ELCQ results to identify controls deficiencies in each of the five COSO components. For this purpose, the existence of any question that received three or more negative responses was presumed to indicate that the underlying control might not be operating as intended. For this analysis IEG conducted a very detailed examination of all questions in the ELCQ that received three or more negative responses, together with the reasons given for the “no” responses and the types of units giving the responses. This analysis was cross-checked with the judgments made in compiling the EPRs for each question (as described in Box B.1). In each case a specific deficiency was identified as being uncovered by these responses, and that deficiency was recorded. 20. From the 18 questions containing three or more “no” responses, IEG extracted a summary of what it believes are 25 potential deficiencies suggested by these responses. (Some responses covered more than one issue.) These potential deficiencies are listed by COSO components, because it is useful when considering remedies for deficiencies to know in which part of the framework remedies are needed and what type of actions may be called for. However, the controls framework works in an integrated way, and some controls deficiencies involve issues that cut across more than a single COSO component. The listing of potential deficiencies is given in Box B.2 below. 21. IEG’s evaluation of the entity-level controls, based on the pass rates reflected in the ELCQ results, shows that entity-level controls appear to operate at an effectiveness level above the 92nd percentile, but are accompanied by 25 identified deficiencies. These apparent weaknesses were concentrated in a few areas, but were found in all COSO components. The concentration is evident from the fact that out of 157 ELCQ questions there were only 18 in which three or more negative responses were received, a criterion that IEG used to identify a potential deficiency. These findings were integrated with those from Part I, and consolidated in a “Deficiency Tracker.” A detailed account of how this was done is given in Annex C in Section II. 20 ANNEX B ANALYSIS AND EVALUATION OF THE ELCQ RESULTS Box B.2. Initial List of Deficiencies Identified by IEG from Managements Questionnaire By COSO Components GRAND TOTAL=25 Control Management and Ethical Behavior: Environment  Staff fear reprisal for reporting infringements.  Reported improprieties are not being acted on, resolved in timely manner.  Need wider reporting of disciplinary actions.  Management oversight over project processing and supervision could focus more on controls issues.  Management often slow to respond to QAG, IEG, INT, and IAD recommendations** **Link from Monitoring HR Policies: component  Lack of links to ethics in OPE.  Staff incentives too small to influence behavior, not linked to project performance.  HR needs to address skill mix, staff qualifications.  Job descriptions should emphasize internal controls.  Lack of routine review of access privileges. Risk Assessment Risk Management:  Risk management less developed than risk assessment.  Risk Scan not integrated into strategic objectives.  Weak links between strategic objectives and resources. Fraud Risk:  Risk of fraud and corruption not fully factored.  Project-level risk not fully factored. Control Performance Measurement: Activities  Key performance indicators (KPIs) not fully developed at individual level.  Segregation of duties at times threatened by lack of procurement qualifications in local staff dealing with PR in field offices. Monitoring  Volcker Report findings (now being acted upon) suggest deficiencies in INT structure and function.  Monitoring systems do not all have specific instruments to ensure that operational management takes action on recommendations.  Information requirements of monitoring units sometimes excessive, and overlapping. Information and Information Systems: Communication  Low information technology user satisfaction and poor data quality in some IT systems.  No formal mechanism for identifying emerging IT needs. Communicating on Fraud and Corruption:  Need for improved training on detecting fraud and corruption.  Senior management needs to communicate consistent message on ethical values, fraud, and corruption.  Disaster recovery and business continuity plans not consistent across all regions. Source: IEG extracts from management questionnaire data. 21 SECTION II The Integrated Internal Controls Framework (Combining Parts I and II) Preamble to Section II The objective of the overall review is to evaluate the effectiveness of the internal controls framework, which includes the key controls in the business processes at the transactions level and the entity-level controls that form the COSO framework. Having completed Parts I and II it remains to integrate the findings of both parts to provide the overall evaluation. Management has concluded that the internal controls framework has been shown to operate in a way that—except for certain identified weaknesses—gives adequate assurance that IDA’s operations are compliant with its policies and procedures, and are conducted with due regard to efficiency and effectiveness. In similar fashion, IEG rates the quality (the effectiveness) of the controls framework as satisfactory with qualifications, reflecting evidence that at both the transaction and the entity-levels pass rates were above the 90th percentile, but there were a number of identified weaknesses in the framework. This final segment of the report summarizes all the weaknesses identified (in a Deficiency Tracker; Annex C), and shows how IEG arrived at its final rating of the overall controls framework, including the material weakness (Annex D) and final overall rating (Annex E). 25 Annex C: Integrating Parts I and II: Scope Limitations and Control Deficiencies 1. This annex begins the integration of Part I and Part II and focuses mainly on controls weaknesses. It deals with certain scope limitations that applied to management’s approach to Part I (which would be weaknesses in management’s assessment had they not been addressed in Part II). It describes compliance issues that were postponed in Part I for completion during Part II. And it lists all controls weaknesses identified in both parts of the review: one material weakness, five significant deficiencies, and some 160 deficiencies found overall, and describes how many have been resolved and those whose remedies are still in progress or still to be acted upon. 2. Because the IDA Internal Control Review was conducted in three parts, Part IA, IB, and II, this annex summarizes the status of issues relating to (1) limitations on the scope of management’s work that was reported by IEG after Part I of the review, and (2) weaknesses in internal controls, which are categorized as material weakness, significant deficiency, or deficiency, that have been surfaced by management, IAD, and IEG in the whole review (both Parts I and II). Scope Limitation Issues Addressed 3. In its Part IB report, IEG had recommended that management do additional mapping and testing of certain processes and key controls to better support its conclusions. In addition, at that time management had postponed some work on the effectiveness of compliance controls at the transaction level from Part I to Part II. 4. As detailed below, work completed by management in Part II has addressed the scope limitations noted by IEG at the end of Part I. ADDITIONAL CONTROL TESTING COMPLETED 5. In response to IEG’s recommendations, management in Part II mapped and tested four additional IDA business processes that it had not assessed in Part I. The additional work did not disclose any control deficiencies in three of the four processes. However, testing of AAA/ESW controls revealed a significant deficiency that management has included in its Part II report and listed in its Deficiency Tracker, along with plans for remedial action. 27 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES AAA/ESW 6. In Part II, management added a new business process module (No. 33, ESW Reports), prepared a process map, and identified four key controls in the processes. However, there is no OP/BP policy for ESW/AAA; rather, the process map and key controls were taken from an “ESW Template for Processing Guidelines.” (We discuss later in this annex that the absence of formal ESW/AAA policy, coupled with other policy-related issues, is a significant deficiency requiring remedial action.)1 Management performed tests for six ESW reports from four regions to determine whether the four key controls were working. Management found noncompliance for three of the four controls and concluded that achieving the control objectives relating to ESW may be at risk. The deficiencies related primarily to insufficient evidence that required steps, including a formal review at the concept and decision meeting stages, were being performed. In addition, in some cases the dates recorded in SAP indicating that certain key steps were performed did not agree with the dates in supporting documentation. In documenting the ESW deficiency, management also cited weaknesses reported by QAG in coding and tracking of AAA outputs and in the managerial oversight of the AAA program. DEBT SUSTAINABILITY ANALYSIS 7. In Part I, management limited its compliance testing to debt reporting and did not include a review of how country debt data were analyzed and used within the Bank for resource allocation and lending decisions. To address this limitation, management added in Part II a new business process module (No. 34 Debt Sustainability Analysis), prepared a process map, selected eight countries for review, and performed tests to determine if required documentation was available for the three key controls identified. The additional review revealed only one exception (evidence of the Regional PREM Director’s clearance of the analysis for one country was not available), and management concluded that the controls reasonably assure that IDA objectives for debt sustainability analysis are being achieved. IDA CREDIT/GRANT MIX 8. In Part I, management assessed the FRM allocation processes except for determining if IDA countries’ financing terms were based on the “balanced perspectives of independent Bank entities (PREM and FRM) and feedback from regional management.” In Part II, management mapped the credit/grant mix process, and tested compliance with the one key control identified (finalize the “traffic lights,” or credit/grant mix) for 10 countries. Management found no exceptions and concluded IDA objectives were being achieved. SAFEGUARD RISK OVERSIGHT 9. In Part I, management concluded that this process (No. 29 – Safeguards – Corporate Risk [QACU]) was deficient because it was not documented and therefore could not be mapped and tested. Since that time, and after IEG’s recommendation (No. 4 in the Part IB report), the process has been documented and management then mapped the process and tested the one key control identified (QACU and Region meet to discuss portfolio and/or individual projects) for a sample of 11 projects/countries. Management found no exceptions and concluded that the IDA objective (to ensure that projects with high safeguards risks 28 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES receive adequate corporate oversight and support in addressing the risk) was being achieved. OTHER TRANSACTIONS TESTING COMPLETED 10. During its Part II work, management also expanded the scope of its testing in two other areas:  QAG – Quality at Entry and Supervision. During Part II, management arranged with IAD to include testing of the QAG QSA 7 process as part of IAD’s planned audit of QAG. Management selected the sample of projects, and IAD performed the testing. All of the eight projects selected and tested were found to be compliant with the six key controls identified, and management concluded that the IDA objective of assuring quality of supervision was being achieved. However, other work performed by IAD in Part II on entity-level controls led to different conclusions about the effectiveness of QAG as a central control unit. (See Table C.2.)  CAS, Development Policy Lending (DPL), and Corporate Review Processes. In Part I, IEG noted that in some cases, Management had limited its testing to three or four of the Regions and questioned the adequacy of the coverage for judging compliance with the key controls identified. To address this limitation, in Part II management added a region (MNA) for testing CAS products (Business Process Module No. 4) and a region (SAR) for DPL testing (BPM No. 7). Management also tested the controls for Corporate Review (BPM No. 8) for these additional regions. Management found no exceptions in any of the three processes, and its conclusions from Part I remained the same. Compliance Issues Postponed to Part II 11. During Part II, management addressed several areas that had been postponed from Part I—namely, IT system controls, transaction controls in use at the field office level, and controls relating to fraud and corruption. INFORMATION TECHNOLOGY CONTROLS 12. During Part II, management determined that the IT testing it had done as part of the ICFR was adequate for the IDA 14 internal control assessment and did not do any further testing. However, several systems excluded from the ICFR work are used or planned for use in IDA operations. Because some of the systems were not fully implemented, they could not be tested during Part II, and management plans to test them when they become operational. IEG agrees that the effectiveness of IT controls in systems such as RAPMAN-PRIMA2 and the procurements complaints database still need to be tested. FIELD OFFICE CONTROLS 13. During Part II, rather than conducting on-site field office work, management elected to rely on the work of IAD’s audits of internal controls over decentralized operations at the field offices. IAD had not synthesized and reported in summary form the results of its numerous audits at country offices but has now done so. Consequently, any systemic weaknesses that may have existed and their underlying causes could not be readily identified and addressed by 29 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES management. Therefore, for this review, management summarized the results of IAD audits at country offices during fiscal year 2007 and the first quarter of fiscal year 2008. Of the 21 audits completed during this period, IAD rated the operations in one report unsatisfactory, 14 as needs improvement, five as satisfactory, and one report did not have a rating. Management noted three areas of weaknesses in the audit reports as Table C.1 shows. Table C.1. Weaknesses Identified in Twenty-One IAD Country-Level Audit Reports Issued July 2006 through December 2007 Area of Weakness Country Office Reports Where the Weaknesses Were Found Number of Reports Percentage Project supervision, especially project supervision reporting 15 71 Procurement process and controls 14 67 Financial management process and controls 9 43 14. The results of IAD’s audits are consistent with the results of other work done in Part I and Part II of the review, IEG has reported significant deficiencies in all three of the areas indicated in Table C.1. However, it should be noted that the institutional learning from the findings of these audits would have been greater had they been analyzed earlier by IAD in a synthesis report, where common themes and their materiality could have been identified. FRAUD AND CORRUPTION 15. Management agreed that in Part II it would more specifically address issues of fraud and corruption. In its Entity-Level Control Questionnaire (ELCQ), management added a set of questions dealing specifically with fraud and corruption, which IEG has analyzed along with all other results of the ELCQ and has presented the results in Annexes B and D of this report. In addition, during Part II management identified those of the processes (mapped in Part I) that had tested key controls that would help in preventing or detecting instances of fraud and corruption. Internal Control Weaknesses 16. IEG reported at the completion of Part I that the conclusions reached on the strength or weakness of the transactions controls were tentative and needed to be validated based on additional transaction testing to be done in Part II and the results of the Part II entity-level review. Final conclusions could not be reached after Part I because weaknesses found in the transaction-level controls could be either mitigated or magnified by the Part II findings. 17. At the end of Part I, IEG had concluded there were one potential material weakness, two significant deficiencies, and three other categories of deficiencies that did not rise to the level of a significant deficiency. After completing Part II, the status of the Part I findings is as shown in Box C.1. 30 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES Box C.1. Part I Findings Subject to Review and Modification in Light of Part II Findings Material weaknesses Significant deficiencies Status after Part II findings Lack of currency of OP/BPs* Remains an issue, but based on improvements noted during Part II, the potential material * Potential material weakness weaknesses was downgraded to a significant deficiency Timely accessibility of operational Remains unchanged, pending development of documents new IT programs Regional variance in dealing with FM and Findings from IAD audits, India and the other procurement procedures DIRs, contribute to an overall material weakness in controls over fraud and corruption Status of Other Deficiencies Noted in Part I Need for improved general management Combined evidence from Part I, from IAD oversight of project processing and audits, from several Part II responses, from supervision QAG ISR rating, from India DIR suggests still a deficiency Need for improved procedures for placing New structuring of POCQC procedures, projects on the Corporate Risk List documentation and review procedures, suggests downgrading to “resolved” Need to extend COSO framework to improve Part II reveals imbalance between risk risk management assessment and risk management; suggests upgrading to significant deficiency OVERALL STATUS 18. During the overall IDA internal control review, a total of 175 individual deficiency issues were recorded in a Deficiency Tracker, including those first identified by management, IAD, or IEG. The status of the deficiencies was as shown in Figure C.1. Of these individual deficiency issues, many have been combined to determine whether they rise to the level of material weakness or significant deficiency. Figure C.1. Status of Deficiencies as of September 2008 250 175 200 NUMBER 150 69 81 100 25 50 0 Resolved Under A ctions Planned Total Implementation 19. The 69 deficiencies in the “resolved” category were initially identified as deficiencies but were later disposed of with no remedial action taken or required, often after additional information was obtained or, in some cases, after further testing of key controls was completed. Management has taken remedial action on 25 of the individual deficiency issues and implementation of new or revised policies, procedures, or other controls to address these deficiencies was underway (as this report was completed) and, in some cases they may have 31 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES been completed but verification that the action has corrected the underlying weaknesses remains to be done. For the remaining 81 deficiencies, management has developed plans and schedules to strengthen the processes and controls but has not begun implementing the changes. 20. Although each of the deficiencies is recorded in the Deficiency Tracker, management decided—and IEG agreed—that many of the deficiencies represented similar or recurring kinds of weaknesses and could be grouped together for reporting and remedial action purposes. Management, IAD, and IEG each made judgments about the level of control risk associated with the deficiencies and classified each as a material weakness, significant deficiency, or deficiency. Management reported five significant deficiencies and IAD reported six significant deficiencies and four issues that require further consideration. IAD further opined that the significant deficiencies relating to fiduciary controls, entity-level controls, IT controls, and fraud and corruption controls, in combination, will become a material weakness if not corrected. Based on the overall results, IEG is reporting one material weakness, six significant deficiencies, and 22 deficiencies. Table C.2 summarizes these issues by management, IAD, and IEG. Table C.2. Summary of Material Weaknesses and Significant Deficiencies Reported by Management, IAD, and IEG Level of deficiency reported by Category Management IAD IEG Fraud and corruption SD SD MW Policy and procedural framework for investment lending SD SD - Lack of focus on key risks and controls SD - Policy framework, outdated OP/BPs SD - Project supervision and diffuse accountability SD - Document retention and accessibility SD Procurement and financial management SD SD SD - Consistency of regional quality arrangements* - Lack of clarity in accountability* Identification of systemic risk SD SD (note 1) Other issues SD - AAA SD (note 2) - IT controls SD SD 6 SDs, 1 MW, TOTALS 5 SDs (note 3) 6 SDs Number of Deficiencies (note 4) 22 Legend: MW - Material weakness; SD - Significant deficiency * These issues are part of an overall SD but are not SDs in themselves. Note 1: IEG did not identify the identification of systemic risks as a separate SD, but rather included these issues as part of the SD relating to lack of focus on key risks and controls identified above. Note 2: IEG does not regard AAA as a separate SD because IEG considers the deficiency to be a part of the broader SD listed above relating to a lack of up-to-date OP/BPs. AAA does not have policy guidance in an OP/BP that requires specific steps of a kind that could be viewed as a key control to ensure AAA performance and quality. Rather, the guidance is in the ESW Template for Processing Guidelines rather than a required policy. Note 3: IAD concluded that the SDs related to fiduciary controls, entity-level controls, IT controls, and fraud and corruption controls, in combination, represent a potential material weakness. IAD also stated that four issues—governance issues, governance and accountability for integrated risk management, oversight of operational risk, and QAG assessment—require further consideration. In addition, IAD stated that management’s conclusion with regard to effectiveness and efficiency of operations is inconsistent with the original objective. Note 4: Numerous individual deficiencies requiring remedial action were identified and are not detailed above but they are all listed in the 32 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES Deficiency Tracker. Many are sub-issues of the above issues and they include the 22 new issues uncovered during Part II (see para. 25). 21. Management should take action to fix these internal control weaknesses with target dates for the actions to be completed. Following completion of the actions taken, management should then test to determine whether the actions taken have indeed resulted in controls that are working to prevent these internal control weaknesses. Once controls over an issue have been implemented and tested and found to be working, this issue can be considered closed. MATERIAL WEAKNESS 22. IEG found (as detailed in Annex D) that there was a weakness in controls over the detection and prevention of fraud and corruption (F&C) in the Bank’s overall operations cycle management, a weakness which, because of the risks that diversion of funds could impair IDA’s mission, is deemed to rise to the level of a material weakness. There were two broad causes for the appearance of this weakness: first, a group of factors, including some deficiencies in the Bank’s generic financial management and procurement controls, a lack of clarity as to the priority of combating unethical practices in Bank operations, and some deficiencies in project supervision and management oversight; and second, the absence of any specific tools to combat F&C in the Bank’s lending management processes. SIGNIFICANT DEFICIENCIES 23. What follows in paragraphs 23-30 is a record of the various significant deficiencies and deficiencies that have been uncovered in both Part I and II. There is some degree of overlap in this presentation, which is difficult to avoid, but the merit of the section is that it does provide a complete record in a single place. In all, at the conclusion of both Parts I and II of its evaluation, in addition to the material weakness described above and in Annex D, IEG found six significant deficiencies: 1. The need to maintain the currency of the OP/BPs (downgraded from potential material weakness in Part I); 2. The need for improved systems of document retention and accessibility (also downgraded from potential material weakness in Part IA); 3. Generic weaknesses in controls over FM and PR processes (identified in Part I) (this also contributes to the material weakness related to fraud and corruption); 4. A need for improved management oversight of project processing and (most particularly) project supervision; 5. A need to improve risk management, including inserting specific F&C risk factors into the Risk Scan, and in moving risk treatment from the entity level to the activity level; 6. A lack of a rapid review of electronic systems access privileges required by staff re- assignments; and a need for enhanced information security and better change management relating to the Bank’s automated systems. DEFICIENCIES 24. IEG found that at the end of Part II, besides the material weakness and significant deficiencies listed above, 22 deficiencies were identified in addition to those identified in 33 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES Part I. The status of Part I deficiencies that were carried forward from Part I is discussed below, following the discussion of Part II deficiencies. Part II Deficiencies 25. Following are 22 deficiencies identified by IEG during Part II, listed by primary COSO component where they were identified or where they should be addressed. Many of these contribute to the material weakness and significant deficiencies identified above. Control Environment 1. Emphasis on Ethical Values: The importance of integrity and ethical values is not well reflected in staff’s performance evaluation. There is a lack of specific mention of ethics in the OPE, where this issue is left to the discretion of the managers’ practice. 2. Anti-Fraud Programs and Controls: Concern has been expressed that staff fear reprisal for reporting infringements and unethical behavior. This deficiency is being addressed by the new whistleblower mechanism. 3. Resolution of Improprieties: Reported improprieties are not followed up on and resolved in a timely manner (identified in the Information and Communications component). 4. Staff Incentive Systems: Incentives do not link to ethical behavior, and several respondents stated that the incentives are too small to influence behavior. There are no real links in the incentive system between staff rewards and project performance. 5. Reporting of Disciplinary Actions: Actions against outside parties (such as procurement debarment) are frequently and widely reported, but disciplinary actions against Bank staff are reported much less if at all. Management has accepted a recommendation of the Volcker report to correct this. 6. HR Policies and Skills Mix and Staff Qualifications: Cases occur on occasion— decentralized procurement in field offices was cited—where less than fully qualified staff members have had to make decisions beyond their level of competence. Maintaining an adequate skills mix for the Bank was identified by the 2006 Risk Scan as one of two high-priority strategic risks (identified in the Control Activities component). 7. Job Descriptions: The fact that job descriptions do not sufficiently define internal controls responsibilities appears as a weakness in the entity-level assignment of responsibilities which may have contributed to the non-compliance with some key controls uncovered during the Part I transaction-level review. 8. Separate Evaluations: QAG, IEG, INT, and IAD all find that management often fails to take timely actions to follow up on audit, investigatory, and evaluation findings (identified in the Monitoring component). Risk Assessment 9. Risk Assessment and Risk Management: IDA has well-articulated risk assessment processes, but is less well structured in risk management (this contributes to the risk management significant deficiency listed above). 10. Risk and Strategic Objectives: Management may not always give sufficient attention to integrating the findings of the Risk Scan process with strategic objectives 34 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES contained in the Strategy and Performance Contract (SPC) (this also contributes to the risk management significant deficiency listed above). 11. Fraud and Corruption Risk: Given the corruption that is endemic to the environment in many Bank and IDA client countries, the Bank may not be placing sufficient attention on the fraud and corruption risk in its strategic planning (this contributes to the fraud and corruption material weakness discussed above). 12. Project Level Risk: In its review of projects, QAG finds risk assessment at the project level to be improved but more needs to be done. During the transactions level evaluation IEG found that at the business process-level risks were not differentiated as to type, magnitude, and probability of occurrence. 13. Links between Strategic Objectives and Resources: Resource allocations are sometimes not closely enough linked to emerging strategic objectives, resulting in a lack of clarity whether certain objectives can be met. Management acknowledges that more flexibility could be helpful in redeploying resources to emerging priorities. Control Activities 14. Performance Indicators: The ELCQ raised questions about whether the Key Performance Indicators (KPIs) were also being applied at activity and individual levels, and whether performance measurement was functioning as intended. 15. Segregation of Duties: In some cases—mostly in smaller field offices—there has been a breakdown in the segregation of duties principle because there were too few qualified staff to share these duties. Monitoring 16. The Volcker Report and INT: The 18 recommendations relating to the structure, reporting lines, and modus operandi of INT have been accepted by management and are being acted upon. 17. Recommendation Follow-up: Monitoring systems do not all have specific instruments to ensure operational management takes action on recommendation. Information and Communication 18. IT User Satisfaction: There is a lack of user satisfaction with IT systems such as SAP and IRIS and it is often difficult to get consistent aggregated numbers in the Bank’s aggregate reporting processes because of information gaps, including in client countries. 19. Identifying new IT Needs: There is no formal mechanism for identifying emerging IT needs. 20. F&C Training: There is a need for improved training on detecting fraud and corruption. 21. Communicating Ethical Values: There is a need for senior management to communicate a consistent message on ethical values, fraud, and corruption. 22. Disaster Recovery: The Bank’s disaster recovery system and business continuity plan is not consistently applied across all regions; it needs to be updated and requires external expertise to broaden it beyond just IT. 35 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES PART I SIGNIFICANT DEFICIENCIES AND DEFICIENCIES 26. The significant deficiencies (and potential material weaknesses) identified by management, IAD, and IEG in Part I were: 1. The inability to provide timely access to documents—considered a potential material weakness by IEG in Part IA. 2. The extent of variances in regional implementation of institutionally endorsed financial management and procurement guidelines. 3. The status of OPs and BPs—considered a potential material weakness by IEG. 27. These significant deficiencies are included in the significant deficiencies resulting from the completion of Part II, above. 28. Deficiencies tracked by management at the end of Part I, and agreed to by IAD and IEG were: 1. The need to streamline investment lending operations 2. The disparity in corporate review between investment lending and development policy lending 3. The lack of timely updates to the Loan Administration System 4. The inconsistency and lack of follow-up in clearing review comments 5. The need for improved controls over the safeguards Corporate Risk List 6. Noncompliance of IDA countries with quarterly debt reporting requirements. 29. Of the above six deficiencies, the need for improved controls over the Safeguards Corporate Risk List has been addressed, as indicated above in the additional work management performed in addressing module 29. The improved controls were implemented and management tested them and found no exceptions. The other five continue to be deficiencies that management has action plans to address, generally included as sub-issues under Part II deficiencies, although about half of the sub-issues supporting these deficiencies have been addressed and actions are in progress. 30. In addition to the above three significant deficiencies from Part I (which were aggregated from 32 individual issues in the deficiency tracker) and the six deficiencies (which were aggregated from 19 issues), 11 other issues and exceptions were included at the end of Part I in management’s tracker. These other issues related to allocation procedures (2 issues), project changes (2 issues), QAG (4 issues), refunds (1 issue), and safeguards (2 issues). At the end of Part II, of the 62 deficiency issues from Part I, 27 issues were closed or have actions in progress, 32 were included as sub-issues of Part II issues, and three were included as open issues from Part I. PART I EFFECTIVENESS AND EFFICIENCY CONTROLS ISSUES 31. At the end of Part I, management had set aside 44 issues related to controls over effectiveness and efficiency to be addressed during Part II. Of these, 19 issues have actions in progress to address them, 21 issues have been included in the Deficiency Tracker as sub- issues of the 20 Part II efficiency and effectiveness controls issues with action plans to address them, and one was included as an open issue from Part I. 36 ANNEX C INTEGRATING PARTS I AND II: SCOPE LIMITATIONS AND CONTROL DEFICIENCIES 32. Thus, management’s Deficiency Tracker at the end of Part II has 24 issues, 20 from Part II, including sub-issues from Part I, and four carried over from Part I (three compliance issues and one effectiveness and efficiency issue). Annex C 1. IEG regards this deficiency to be a part of the overall significant deficiency relating to areas of the OP/BPs which are in need of updating or formulation, and does not count it as a separate significant deficiency. 2. RAPMAN = Risk and Portfolio Management System launched in MNA and ECA in 2004; PRIMA = Portfolio and Risk Management System launched in AFR, EAP, LCR and SAR in 2007. 37 Annex D: Factors Combining to Form a Material Weakness Introduction 1. This annex provides a detailed account of the background, evidence, criteria, and judgments relating to the IEG conclusion that weaknesses in the complex of controls that govern IDA’s efforts to ensure against F&C in its lending operations rise to the level of a material weakness. Management has itself identified the same set of weaknesses, but has concluded that their materiality represent a significant deficiency (as does IAD), which is being addressed by several initiatives now underway. There is therefore agreement on the nature of the weaknesses, but a difference in judgment as to their materiality. IEG sees the risk of F&C in the context of IDA operations as a fundamental dimension of the development challenge, but believes it would be premature (from both process and substance points of view) to conclude that F&C risks have been successfully tackled under the current IDA controls framework. 2. As background, it is important to bear in mind that for the past decade or more the Bank and IDA have taken the lead and have been actively engaged in addressing issues of governance including the potential negative impact of fraud and corruption on the development programs in its client countries. As is well described in the management assessment, and as is summarized in the following section of this annex, these initiatives have been robust and visible, they have included both global and country level focus, and have involved the development within the Bank Group of new entity level mechanisms to address F&C issues on a broad front, including the possibility of F&C affecting projects and programs supported by Bank and IDA funds. What has been brought to light by this internal controls review is that these mechanisms require further strengthening, and in particular that additional, specific transactions level controls to detect and prevent F&C in IDA operations are needed. Management has vigorously responded to this finding and, as described in its report, is engaged in developing new systems and practices, though it will necessarily take some time for these systems to be implemented and to become fully operative. 3. This annex provides the detailed rationale for the identified material weakness. It divides into four sections: Section I credits the Bank for recent major fraud and corruption (F&C) initiatives but points out that the internal controls to make these effective are not yet in place. Section II summarizes evidence of deficiencies in the existing controls that further detract from the F&C agenda. Section III argues that the combination of these two sets of weaknesses constitutes a material weakness. Section IV suggests the types of remedies but these should be read as indicative, since the scope of IEG’s evaluation precludes detailed prescriptions for corrective action. 38 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS I. The Bank’s Fraud and Corruption Initiatives: The Need for New and Innovative Approaches to Control 4. Since President James Wolfensohn decided in October 1996 to place corruption openly on the Bank’s agenda, sound governance and a focus on corruption in its client countries have been made an increasingly explicit part of the Bank’s, and thus IDA’s, program. The key elements of its efforts to confront F&C issues in its operations have included the following:  High-level speeches: Successive Presidents of the Bank have made statements to the global audience, as well as to the staff, about the importance of combating F&C because of its debilitating impact on the development agenda, and emphasizing the Bank’s commitment to integrity as being at the core of all that it does.  Major Reports: Since the time of President Wolfensohn’s speech to the 1996 Annual Meetings, governance and accountability have been on the agenda of the World Bank Group. Main reports included the 1997 Helping Countries Combat Corruption: The Role of the World Bank; the 1997 World Development Report, the State in a Changing World; and the 2000 strategy paper, Reforming Public Institutions and Strengthening Governance.  Analytical Programs: Several country reports have been written for local stakeholders (government at all levels, civil society, and academia) to draw attention to F&C issues in a range of countries, raise awareness and suggest remedies. Many of these programs identified lessons and experiences also at the country level.  Establishing INT: Set up as an investigatory unit in 2001, INT has become a key element of the Bank’s F&C monitoring and investigation system; it has now conducted six F&C focused studies: Three Detailed Implementation Reviews (DIRs) (India, Kenya and a narrower DIR on Vietnam) and three Fiduciary Reviews (FRs) (Indonesia, Cambodia and Vietnam). All studies showed significant evidence of F&C indicators. INT is a key element in the Bank’s anti-fraud and corruption program, and, following the outcome of the Volcker Report which reviewed its modus operandi, management has completed actions on 16 out of the 18 recommended steps to improve its operational effectiveness. 5. Launching the Governance and Anti-Corruption (GAC) initiative and Strategy: As a global initiative to heighten awareness, and develop the tools needed to combat F&C in Bank and IDA, operations management launched the GAC program in late 2006. Its presentation Strengthening World Bank Group Engagement on Governance and Anti-Corruption lays out a number of principles and concepts as guidance to the measures the Bank needs to adopt in building its portfolio of instruments that need to be developed in tackling the program. In March 2007, the Board of Executive Directors endorsed a new GAC strategy, which calls for the Bank Group to step up the inclusion of appropriate fraud and corruption diagnostic and mitigation measures in CASs and lending. The GAC also recommends better dissemination of INT findings and emerging good practice and more explicit training and sensitization of task teams in how to spot “red flag” indicators of fraud and corruption. In August 2007, Operations Policy and Country Services (OPCS) circulated for discussion a draft plan to articulate concrete steps to implement that strategy, and this was discussed at the Board on October 21, 2008. In addition, the FY 2007 COSO report identifies specific 39 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS actions to improve entity-level controls and anti-fraud programs and controls, including a new Code of Professional Ethics for Bank staff and training for staff on internal controls. The report also identified additional remedial actions including the need for central units to work better together on risk identification, responses to risks, and information sharing and the need to strengthen high-level risk management; including reporting key findings periodically and systematically to Senior Management. 6. Sound initiatives, still lacking specific tools: There is no question that the general awareness of F&C issues has been raised significantly both within the Bank and in its client community, as has the consciousness of the staff that the Bank is determined to confront this issue. Global statements from the Bank, specific country studies, and the investigations completed by INT have also all helped to send a message of the Bank’s seriousness in this area. 7. Viewed from the perspective of the internal controls review, IEG observes that initiatives to translate this awareness into adequate specific tools to assist the staff in tackling F&C issues in their ongoing operations are being implemented under the GAC initiative, but are still in early stages. In its investigation of the India health sector projects (released in 2007) INT observes gaps (that is, missing tools to combat F&C in Bank operations or internal controls specifically adapted to F&C issues) at all key levels: at the entity level, at the country/activity level, and at the project level . The GAC program has called for the development of such tools, and some regions have responded to the INT DIR findings with action plans specifically geared to addressing F&C issues. However, specific Bank-wide F&C controls that are needed in several key areas to make this agenda effective are not yet in place, as summarized in paras. 9 and 10. 8. Risk Assessment: At present F&C risk is addressed in the Bank’s annual management assessments of internal controls over financial reporting (ICFR). However, the ICFR process relates only to financial reporting and this does not address issues relating to possible F&C within Bank/IDA lending operations. 9. There are also other areas where F&C risks need to be more explicitly addressed, including:  Risk Scan: The Bank’s annual Risk Scan, which addresses a broad range of risks, has not specifically targeted F&C risk. F&C risk was not included among the 40 items contained in the Bank’s annual Risk Scan (RS) process, though this is now to be done under a revised risk management regime under development.  Risk Opportunity Workshops: CSR, which administers the annual Risk Opportunity Workshops, has confirmed to IEG that fraud risk assessment is not undertaken in a specific systematic manner. Given the endemic F&C environment in many countries in which the Bank and IDA operate, this is a significant omission.  INT Comments: In a meeting with IEG, INT emphasized that the Bank Group in general needs to develop a better understanding of the specific risks as relating to its instruments, country engagements, sector efforts, and projects and it also plans to continue its support of operational risk assessment. 40 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS 10. Outside of risk assessments, the treatment of F&C considerations has often been sparse, although it has now begun to be addressed better in several important documents and processes:  Country/ Sector Strategy: The CAS and Sector Strategy processes have not systematically and seriously addressed fraud and corruption risk at the country level. Management is now trying to change this, and under the CGAC being undertaken as part of the GAC initiative it should become routine for a CAS to contain a section on country governance, which would often include F&C issues. The CAS is clearly an instrument which could take the lead in setting the stage for F&C risk assessment, as a backdrop to addressing transaction level F&C risks in the projects of a given country.  Project Design: The PAD has historically had no requirement to address a project’s risk of F&C as part of the routine project preparation, and project design does not consider F&C issues specifically and systematically. In some of its reports INT has cited project design as one weakness in addressing F&C risk. However, the GAC progress Report1 has described efforts to improve the attention to F&C issues in project work, also while trying to ensure that such work does not focus exclusively on fiduciary issues alone.  Project Supervision: The ISR has until now not contained any F&C section and no requirement to reflect F&C issues in any existing components (for example among the reported risk flags). By the same token, Task Team Leaders (TTLs) have not been given toolkits or training to address these issues explicitly. These elements are in the process of being addressed under the “GAC in Projects” program, but their scope and eventual effectiveness remain to be verified.  Fiduciary Processes: Existing FM and PR guidelines were not specifically designed to address F&C risk, or prevent the occurrence of F&C, although if properly applied they should implicitly do so. In particular, the Bank’s Procurement Guidelines were designed to ensure equity and economy, and there is no explicit F&C prevention in these guidelines. Some recent INT findings in Bank projects showed that F&C indicators were found even though Bank procurement procedures had been correctly followed, suggesting further that, even where normal procurement procedures are followed, there is a need for more explicit adaptation—either of new and specific F&C related controls, and/or improved application of controls by project staff—to combat fraud in procurement processes.  Country systems: Part of IDA lending is disbursed in the form of general budget support (DPLs/DPCs/PRSCs). These resources enter the general budgets of borrowing countries, and the country’s control systems apply. To meet both fiduciary/control and development goals it is important for the Bank to provide support to the development of those country systems, as it has been doing in a number of ways, including through the Country Financial Accountability Assessment (CFAA) and Country Procurement Assessment Report (CPAR) processes. There remains a need to give attention to the specific safeguards against F&C in these operations, in addition to those measures being taken in the case of ILs. 11. Programs to pursue broadly the combat of fraud and corruption in IDA operations cannot succeed at the operations level without the serious introduction of new and 41 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS innovative approaches to F&C of the kind mentioned above, and which are found in many aspects of the GAC initiative already underway. Specific F&C toolkits need to be devised and locked on to the Bank’s controls architecture adapted to needs at each level. II. Deficiencies in Existing Controls 12. There is evidence from various sources consulted by IEG during Part II that suggests strongly that F&C risk has not been adequately addressed by the existing controls framework. This evidence includes the following: evidence from the ELCQ pointing to some deficiencies in entity-level controls; the DIR/FR studies completed by INT since 2000 which show significant indicators of F&C in a number of sets of Bank projects; the Volcker report, which recommended that INT restructure and reform its role; evidence from QAG assessments and country program audits by IAD suggesting the need to strengthen project supervision; and evidence on significant deficiencies in FM and PR uncovered during the transactions-level testing during Part I of the review. The following paragraphs outline the relevance of this evidence to the F&C agenda and suggest how each part may have contributed to the overall material weakness. 13. Evidence from ELCQ—entity-level deficiencies: In the responses to the questionnaire there was evidence of weaknesses in relation to addressing F&C risk in four areas: a below-average simple pass rate (SPR) in responses to questions in the F&C section overall; two elements within the Bank’s Control Environment—tone at the top, and certain aspects of HR policies; and factors related to risk assessment.  Simple pass rate: The responses to questions in the section of the questionnaire dealing with Anti-fraud Programs and Controls achieved a SPR of 92 percent, 3 points lower than the questionnaire average of 95 percent. This is not overly significant but may reflect concerns among respondents that F&C controls are an issue. These concerns related to whistleblower mechanisms and ethical conduct (Q8), F&C risk assessment (Q18), and ethics-related training (Q32). INT was a lead respondent questioning whether sufficient training against F&C is available, and stressing that no “systemic” F&C risk assessment tools are in place, either Bank-wide or in the Regions.  Tone at the top: Respondents to the ELCQ acknowledged that Senior Management of the Bank has spoken clearly (for the past decade) on the need to combat F&C, and this has been reinforced by successive presidents. However, some questionnaire responses (by INT in particular) suggested that there is still fear among some staff that seeking out F&C issues in projects and reporting on observed improprieties may lead to reprisals from their managers, and managerial signals and behavior are not always consistent with these messages. Overall, mixed messages and ambivalence are still considered prevalent.  HR policies and staff incentives: Questionnaire responses suggested that there is a cluster of HR-related issues that create disincentives, or do not specifically reward the adoption of an F&C agenda. For example, as stated in the responses, there are no explicit links to ethical behavior and no focus on F&C issues in the OPE criteria. Staff are rewarded more for delivery than for behavior and are not encouraged to focus on F&C in Bank projects and other operations – further evidence of mixed explicit or implicit messages to staff on this issue. Staff performance ratings are not linked to 42 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS project performance. Training on the skills needed to detect F&C is in place, but it needs to be deepened and widened (See BoxB.3 in Annex B).  Risk assessment: A number of questions in the Risk Assessment segment of the questionnaire addressed issues relating to F&C. Most units responded in the positive to the question whether the Bank considers fraud as part of its enterprise risk assessment (Question 15). However, INT and others took it as a “serious omission” that the Bank’s Risk Scan omitted F&C risk. Other questions were mostly indirect and not explicitly geared to F&C. For example, Question 57 asked whether IDA considers risks arising from external as well as internal causes, INT referred in its response to the fact that many countries had low capacity to deal with corruption. Question 58 asked whether the Bank learned from past failures and INT stated that IDA did not always do so. The questionnaire responses tended to support the finding that risk assessment of F&C is not strongly integrated into overall risk assessment. 14. Evidence from INT: F&C risk is significant—did controls fail? The critical evidence that has emerged is from several DIRs/FRs that have been completed by INT since 2000, all of which have involved IDA countries, and all of which revealed significant indicators of F&C in the reviewed projects. This evidence has included indicators of collusion on bidding, over-pricing of bids, bidding with fictitious entities, delivery of substandard civil works or equipment supplies, and the paying of bribes to officials . Variations in the patterns of these indicators have appeared in all the countries examined in the DIRs. IEG makes the following observations regarding these findings:  Finding indicators of F&C does not constitute proof that F&C has in fact occurred. However, INT states that there appears to be a correlation between F&C and actual occurrence of the latter and cites subsequent convictions, debarments, and other enforcement actions from some DIRs to support this relationship, including the fact that criminal investigations are ongoing in 13 bribery cases in India.  In India, IEG observes that the DIR has led to quick and significant reforms by government agencies in areas where indicators of irregularities have been found, suggesting there has been substance to the concerns raised.  The types of F&C that may have occurred—collusion, over-pricing, under-delivery, bribery—all imply that there may have been a diversion of IDA funds from the purposes intended in the given projects. If this has occurred to a significant extent, this would then imply impairment of the IDA mission, since IDA funds would have been used for purposes not in compliance with intended objectives, policies, and procedures, and would thereby also impair the efficiency and effectiveness of IDA operations.  Such occurrences, if true, would not be inconsistent with nor would they have been brought to light by the Bank’s annual, unqualified financial reporting and auditing process because the latter considers financial reporting rather than the Bank’s operations. 15. The presence of widespread indicators of F&C, whether proven to be actual or not, do constitute evidence of a significant risk of F&C. The question is, therefore, which internal controls over Bank operations may not have operated effectively—or have been absent—so that these occurrences were not prevented? Part of the answer to this question is to be found 43 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS in the fact that the specific controls and approaches that were described as being needed in paras. 9 and 10 are in fact missing or are only now being put in place. A second part of the answer also lies in some of the deficiencies that have been found in other parts of the controls framework, brought to light by the questionnaire and the other sources of evidence, as described below. 16. Evidence from the Volcker report—role and function of INT: The Volcker Panel observed that the importance of the Bank's internal anticorruption effort extends well beyond the immediate fiduciary concerns, and that the Bank's support for national efforts to improve governance must be at least an equal part of its own efforts to deal with corruption in programs that it supports with its own funds. Ignoring the issue, or, more subtly, tacitly supporting superficial government efforts where there is little political commitment, would conspire against aid effectiveness and the welfare of a country's poor. The Panel further commented that attention to risk assessment and risk abatement strategies must run right through the organization, drawing on expertise in procurement, disbursement practices, and institutional analysis. 17. Against this background, the Panel found that since its inception INT has provided useful information on the presence of F&C risk and the modalities of potential F&C in Bank projects. In some aspects of the way INT has functioned in the past, however, the dissemination of this knowledge has been less than it might have been, and this may have hindered the learning that should have followed from these findings. A review of the role and modus operandi of INT was published (the Volcker Panel Report) in September 2007 in which 18 recommendations were made to restructure and reform the way INT works. The recommended reforms have already been accepted by management and are under implementation. Most of the Panel’s recommendations are designed to improve the flow of information and communication among various operating and central control units in the Bank. IEG acknowledges that management’s response to the report has been swift and appropriate. However, in the strict application of audit standards the deficiencies that the report identified must continue to be regarded as deficiencies until their remedies have been shown to be fully operative and effective. 18. Evidence from QAG—project supervision issues (QSA7): As was described in para. 5, project supervision is a critical element in the Bank’s F&C agenda, because it is the process by which the Bank tracks the performance of projects being implemented by the borrowers. Taken overall, the evidence of the QAG assessment from QSA7 (FY05-06) showed that there has been a general improvement in supervision quality compared to QSA6 (FY03 and 04): QSA 7 showed 95 percent moderately satisfactory or better compared to 90 percent in QSA6. These results show a welcome improvement in supervision where performance in the past had been a weaker link in the Bank’s operations. However, within this improving envelope, QAG itself notes “significant scope for further improvement” and two specific quality dimensions relevant to the F&C agenda which were assessed to be among the weakest:  Candor and quality of ISR: This was rated at least 10 percent lower than all other quality dimensions (85 percent moderately satisfactory or better) and in terms of satisfactory or better has declined (from 49 percent in QSA 6 to 35 percent in QSA 7). This suggests that ISRs quite often still do not fully reflect project realities, a 44 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS weakness that would need to be addressed. If the ISR is to have a new F&C toolkit appended to it (as suggested in para. 10 above) this would be of little avail if the ISRs should not to be taken sufficiently seriously by TTLs and reviewing managers.  Supervision budgets: QSA 7 shows that among the items rated among the lowest quality dimensions (78 percent moderately satisfactory or better) was the adequacy of supervision budget, where 20 percent of projects were assessed as having budgets deemed too low. If project supervision will need to carry an increased burden in the fight against F&C, this is going to require more staff time and therefore greater resourcing of supervision budgets. 19. Evidence from Part I—significant deficiencies in financial management and procurement: Where there is a risk of fraudulent and corrupt activities it is generally in relation to the procurement of goods and services and the disbursement of funds to suppliers and others implementing Bank projects. The Bank’s FM and PR controls, therefore, are its fiduciary front line. In fact, during the testing of key controls over the FM and PR business processes during Part I management, IAD, and IEG all concluded that the quite high rates of non-compliance in these modules were such as to constitute a significant deficiency. Also, information gathered revealed two control weaknesses relating to IDA’s handling of procurement complaints. First, there was no control to ensure that all complaints were entered into the complaints database. Having all complaints in the database is the first step in ensuring that all complaints are handled appropriately, such as by being referred to the INT and/or considered for potential noncompliance with IDA procurement policies. Second, there was no control to ensure that reports from the complaints database are followed up for all complaints, because there are no regular reports produced from the database. Because of these weaknesses, the monitoring control that complaints themselves provide to IDA may not function as effectively as it should and problems in procurements may go unaddressed. Subsequent to IEG’s Part I report, INT issued the DIR report on the health sector in India, which revealed numerous indications of F&C in IDA projects and according to INT, the source of these indications in many cases was procurements complaints received by the recipient country government and/or by IDA. Until IDA strengthens the procurement complaint handling process, it runs the risk that detectable F&C will not be caught and dealt with in a timely manner. 20. IAD Country Audits: As was described in paragraph 14 in Annex A, the evidence from a number of IAD country audits that showed some deficiencies in controls over FM and procurement processes in country programs was also relevant, and added support to the finding of generic controls deficiencies. III. A Material Weakness in Controls over F&C 21. Summary: The preceding two sections described a number of internal controls deficiencies—and one significant deficiency from Part I (in FM and PR) which would affect IDA’s and the Bank’s ability to address F&C risk, and thereby assist in the fight against fraud and corruption in its operations. In IEG’s judgment, except for the controls over FM and PR, none of the other weaknesses identified would, on their own, rise above the level of a significant deficiency. The question is whether the combined impact of these various deficiencies would constitute a significant deficiency or material weakness. In examining 45 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS this question, IEG was influenced by the extent of the F&C risk, and by the fact that controls did not match this risk. 22. F&C risks are higher than expected: Until the establishment of INT and the launching of the Bank’s F&C agenda, there was no systematic examination of whether fraud and corruption was occurring in operations supported by IDA. The evidence brought to light by recent DIR and related studies shows that, indeed, the risk of F&C is significant. In view of the development environment in which IDA is working in many countries, this risk may therefore likely apply to a considerable number of client countries and thus constitutes a potential risk of impairment of IDA’s mission, if not adequately addressed. If F&C actually does occur, this could therefore involve diversion of funds on a significant scale across a number of countries, which would impair IDA’s mission, breach compliance with IDA policies, and put IDA’s reputation at risk. However, IEG emphasizes that its conclusion of a material weakness in controls over F&C is based on the evident risk, and not on any evidence of widespread actual F&C. 23. Are F&C risks matched by F&C controls? The essence of an internal controls system is that it identifies priority risk and makes sure that controls are in place to match the risks. As the two earlier sections have described, IEG finds that this is not the case, in two key respects:  As argued above, improving the effectiveness in operation of the existing controls over FM and PR is a necessary condition of addressing F&C risk. However, this would not be sufficient without also adding new approaches to explicitly combat F&C risk.  Controls to adequately match F&C risks in Bank-financed projects need to be set in place not just in the fiduciary processes but at all key stages of the project cycle: country strategy, project inception, project design, and project supervision. Specific F&C toolkits need to be designed to do this. As an extension of this also, the governance and fiduciary systems within the countries themselves (on which the Bank and IDA may place increasing reliance in the future) as well as those used by other donors and partners must also be strengthened to match these new controls within the Bank and IDA, since internal Bank controls will not alone be sufficient to effectively combat F&C. 24. How to judge the materiality of these deficiencies? In making its evaluation and deciding that the weaknesses amount to a material weakness IEG was guided by the agreed standards by which to judge materiality (see Box D.1 below), which were agreed at the beginning of this review. The materiality of the deficiencies in controls over the FM and PR processes has already been declared in Part I to be a significant deficiency. No mitigations were found for this during Part II. Further, several other deficiencies have been identified whose impact would tend to reinforce the weakness in the controls framework in combating F&C. It is, therefore, difficult to avoid the conclusion that, overall, the materiality of this weakness appears greater now than it did at the completion of Part I, and that, if not addressed, could potentially impair the IDA mission. 46 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS Box D.1 Standards and Evidence Which Led to a Finding of a Material Weakness At the start of this review management, IAD and IEG agreed on the standards that were to be followed in determining the materiality of any controls weaknesses that were uncovered during the review. These were presented in Annex B to IEG’s the report on Part IA. Identified deficiencies could be significant deficiencies or material weaknesses depending on whether one or all of the materiality criteria cited below are present. MATERIALITY CRITERIA SPECIFIC EVIDENCE FOUND  Evidence from INT DIRs and FRs in five countries show F&C indicators which suggest funds may have been diverted from the purposes intended; evidence of actual F&C has also been  Impair the found in some of these cases achievement of  INT DIRs/FRs show that F&C indicators appeared in all six IDA’s objectives studied countries, suggesting actual F&C may not be rare but may occur in many countries  IDA’s Articles require that IDA operations be governed by its  Violate policies and procedures requirements of  INT DIR evidence (India, Kenya) shows that F&C indicators IDA’s charters or have been found even when Bank procedures have been other contractual correctly followed agreements  In IL operations evidence from Part I showed significant  Significantly deficiencies in IL fiduciary controls and absence of F&C weaken safeguards controls against waste, loss,  In DPL/PRSC operations procurement controls do not apply, so or unauthorized F&C has to be addressed through IDA FM and Loan use of funds, Administration controls and through country systems which property, or assets bare often weak or non-existent  Involve conflicts of  Bank culture, management priorities, staff incentives and HR interest, involve practices have not given priority to safeguarding against F&C systemic problems in Bank/IDA operations in country  Evidence from Part I (fiduciary weaknesses) and Part II (entity assistance, level incentives) shows a gap between progress in building a partnerships and global Bank agenda against F&C and establishing tools to project lending, or make the agenda operational; more is needed from Senior require the Management in linking the global perspective to daily attention of Senior operations and to internal systems in client countries Management, the Board as well as the awareness of external stakeholders 25. It needs to be emphasized that the audit standards themselves do not dictate the circumstances or criteria by which a distinction is to be made between a significant deficiency and a material weakness. The generic criteria are clearly stated (as shown in Box D.1), but it remains to some extent a judgment call when to designate a particular deficiency 47 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS a material weakness (and in the present review management, IAD, and IEG, while agreeing on the types of weaknesses, all came to slightly different judgments as to materiality). 26. Conclusions: IEG concludes that a material weakness exists in the internal transaction- level controls framework to meet the risks that local beneficiaries, participating suppliers and other stakeholders in IDA projects do not subvert the objectives of IDA lending operations through fraud and corruption. Based on the technical criteria agreed for this review IEG finds that, to varying degrees, all of the attributes listed in Box C.1 are present. This weakness does not arise from a single-point control that is badly designed or is not observed, nor does it relate solely to the Bank’s fiduciary processes (though the latter are centrally involved). The weakness rather derives from a combination of some inadequate assumptions regarding the prevalence or importance of F&C in Bank operations, which emanate from a certain lack of entity-level awareness and propagation (now in the process of being addressed), and - until the introduction of new controls including some now being put in place - a corresponding absence of specific controls to combat F&C in IDA operations and their related country systems, all of this exacerbated by some generic deficiencies in existing controls. IV. Summary and Suggested Remedies 27. Suggestions as to remedies:. Given the composite nature of the material weakness IEG proposes that the remedies would also need to be multifaceted. This fact has been well recognized by management (and by IAD) and is reflected in the wide range of GAC program components currently being implemented or prepared. In the schematic shown in Figure D.1, IEG displays the key elements it regards as contributing to the weakness and shows the types of remedies that could be applied at the entity level and in the business processes across the project cycle. IEG highlights the following elements: a. Tone at the top: By providing the clear and consistent messages from Senior Management emphasizing that adherence to controls to prevent F&C is an explicit part of IDA’s mission, managers would encourage staff to integrate this focus into their ongoing operations at all levels, matching incentives for internal ethical behavior with enhanced external awareness of and vigilance over the potential for F&C to occur. b. Match culture with tools: Developing specific tools to focus on F&C risk at the entity, country, and project levels—and specifically by providing F&C tools in the CAS, PAD, the ISR, and support for country systems—which will also reinforce the cultural factor at the level of the operating staff. c. Tilt in favor of supervision: Provide more emphasis on project supervision as a key element of an F&C agenda. Adopt TTL training and other F&C tools. Where necessary, shift in budget allocations from loan preparation to loan supervision. d. Learn from INT findings: In line with the recommendations currently being worked on by management in its response to the Volcker report, provide greater dissemination of INT findings in ways which will ensure that experience across the Bank leads to improved learning on F&C issues in IDA operations. e. Update FM and PR guidelines: In addition to seeing to some generic FM and procurement controls issues (identified during Part I), there should also be a focus 48 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS on linking these to the F&C measures to be introduced into project supervision to ensure that controls are tightened in both areas. Figure D.1. Remedies to Combat a Material Weakness in Controls over Fraud and Corruption Control Environment •Clear management signals •HR policies, staff incentives •Results-based management Country Strategy •CAS process to reflect F&C country risk •F&C safeguards in DPL/PRSC Risk Assessment Project Design •F&C in Risk Scan •F&C section in PAD from CAS country risk Project Supervision •Implementation Status Report (ISR): F&C toolkit •Adequate budgeting New Specific F&C Tools •INT assist design •Results monitoring •GAC program •OPCS VPUs Financial Management/Procurement Controls •Remedy deficiencies •Add F&C tools 28. Remedies Already Underway: Many of the suggested elements that IEG is suggesting seem to be contained in the work currently being implemented or planned under management’s response to the identified control weaknesses: under the GAC initiative; in the management response to the Volcker Report on the working of INT, and in some of the recently initiated INT outreach and training work The management plans and actions as part of these initiatives seem appropriate and aimed at priority aspects of the controls weaknesses. At the same time, most of these initiatives are quite recent, dating from the completion of the Volcker Report or from the early findings of this review. Many of them are therefore not yet fully operative. IEG therefore believes it would be premature (from both process and substance points of view) to conclude that F&C risks have been successfully tackled under the current IDA controls framework. By this token, while these remedies can be recognized as being well suited to the risks they address, under the COSO criteria which have governed this review, the remedies cannot yet be taken to have removed the material weakness that IEG identified.2 Annex D 1. Governance and Anti-Corruption Report, presented to the Board in October 2008. 2. Under the COSO concept of reporting on internal control system effectiveness, if one or more material weaknesses exist (which would preclude a statement that the criteria for system effectiveness are met), a description of the material weaknesses should be included in the report until 49 ANNEX D FACTORS COMBINING TO FORM A MATERIAL WEAKNESS such time that “management has determined that the new or revised controls were operating effectively.” Relevant auditing standards (AS5 and its predecessor, AS2) similarly state that any material weakness existing as of the date of the report must be disclosed. 50 Annex E: IEG Composite Evaluation of the Internal Controls Framework Description of Approach 1. This annex contains IEG’s final composite evaluation of the integrated controls framework as a whole. It is based on all sources of evidence. This includes taking account of the underlying analysis presented in Annexes A and B (showing the ELCQ results), it integrates findings from both parts of the review, it considers other evidence available and it reflects the results of the Template application that IEG used to rate the effectiveness of the controls system. This is where, based on all sources of evidence, IEG gives its own independent evaluation of the effectiveness of the controls framework, as is summarized in Volume I. 2. IEG’s overall evaluation started with a thorough review of the basic method that management used in its assessment approach, and found that it was sound and was not itself the cause of any serious doubts about the results obtained. IEG also reviewed management’s mapping and role assessment of the Bank units that comprise the entity-level controls framework, and it conducted an extensive analysis of the data that management presented in the form of the ELCQ responses. Together these analyses enabled IEG to formulate its evaluation of the effectiveness of IDA’s entity-level controls, and therefore to complete Part II of the review. Taking the step of integrating the findings from Parts I and II, the question was also asked whether deficiencies identified during Part I would need to be revisited and possibly revised in the light of findings that emerged after completing Part II. Consulting all forms of evidence available for the review, including evidence (such as recent DIR reports from INT) not directly related to the review, IEG was able to formulate its evaluation and provide a final overall quality rating for the effectiveness of IDA’s internal controls framework. Rating the Effectiveness of IDA’s Controls 3. The Entity-Level Template, details of which are given below in paras. 6-16, was the principal tool IEG used to rate the overall controls framework. The template contained a set of key questions regarding the effectiveness of controls within each COSO component, in light of all the evidence assembled. This included the evidence of a material weakness and six significant deficiencies that IEG has identified by the end of Part II, together with all the evidence of compliance from Part I and of entity-level controls pass rates in Part II. Once completed, the template provided a rating for controls effectiveness within each COSO component. These were then mapped onto a statistical “diamond,” as shown in Figure E.1. 51 ANNEX E IEG COMPOSITE EVALUATION OF THE INTERNAL CONTROLS FRAMEWORK 4. The effectiveness of controls within each component was rated on a scale of 1 (fully satisfactory) to 4 (unsatisfactory). The figure shows the quality ratings for controls within each COSO component as one arm in the diamond. Had controls been evaluated as fully satisfactory without qualifications in all cases (rating 1), each arm of the polygon would lie at the outer perimeter. In actuality, after examining all the evidence, IEG found a material weakness and six significant deficiencies in the controls framework (see below). It found further that that these weaknesses cut across several or all components in the COSO framework. As a result, using the IEG rating template, even though the controls within each COSO component were rated separately, the result was that each one was rated satisfactory with qualifications. Hence, the results line in the figure lies symmetrically along the rating line 2. Figure E.1. IEG’s Evaluation of the Effectiveness of IDA’s Internal Controls Framework Control envrionment 14 3 2 Information and 2 3 Risk assessment communications 1 4 Monitoring Control activities 5. This depiction accurately expresses IEG’s overall finding, but as a statistical aggregate, it also masks some important features that need to be highlighted. One is that, as the analyses in Annexes A and B show, the more serious weaknesses in controls—even though the cut across COSO components—were concentrated in a relatively few areas in the controls framework. Apart from these weaknesses, the framework operates effectively. In the same way, IEG’s 2 ratings for the controls within each component, while valid overall, hide the fact that many individual control items were rated 1 or fully satisfactory. Out of 74 items rated, 43 (58 percent) were rated 1. (See Statistical Appendix Table SA.7.) 6. The finding of satisfactory with qualifications is similar to management’s findings that the controls framework offers “adequate assurance to Senior Management, except for 52 ANNEX E IEG COMPOSITE EVALUATION OF THE INTERNAL CONTROLS FRAMEWORK certain deficiencies.” The evidence that the framework is “satisfactory or adequate” was supported by the pass rates calculated in both Parts I and II, which were above the 90th percentile—a 93 percent pass rate measured by compliance at the transactions level (in Part I), and 92-95 percent pass rate measured by ELCQ responses at the entity level (in Part II). Regarding evidence for the “qualifications” or the “except for” aspects of the controls, these are evident in the one material weakness and six significant deficiencies that were uncovered during the review. Using the IEG Template to Rate the Controls Framework 7. In completing its evaluation IEG used a combination of analytical tools, including the statistical analysis of the ELCQ data, review of management’s evidentiary materials and its final report, interviews with most central control units in the Bank, reviews of various background papers, and a review of the IAD final report. In addition, IEG used two especially designed templates to combine the evidence from all these sources and methods to derive a single rating for the overall quality (effectiveness) of the controls in question. Two templates were used because the needs of Part I and Part II were different:  In Part I IEG used the Business Process Template, which was designed to standardize IEG’s analysis and rating of each of the 30 Business Process Modules (BPMs) that management mapped and tested during its transactions-level assessment in Part I. Hence, the template was applied 30 times, once to each BPM. The results were reported in IEG’s Part IB report.  In Part II IEG used the Entity-Level Template to rate management’s and IAD’s approaches and results in their review of the entity-level controls. This template was used also to generate IEG’s overall ratings of the controls effectiveness in the integrated framework as a whole. The Entity-Level Template was used four times, as described below. 8. Content: Each template has three main elements:  A set of standardized questions aimed to provide IEG with a rigorous approach to evaluating management and IAD approaches and results.  The rating system was designed to provide a standard metric with which to rate the results of IEG evaluation.  A topic structure organized the questions and ratings by topics, permitting the template to be used four times to rate the following aspects of the evaluation: 1. Evaluation of Management’s Approach and Method; 2. Evaluation of IAD’s Review and Opinion; 3. Evaluation of content by COSO components (that is, IEG’s final overall rating of the effectiveness of IDA’s internal controls framework); and 4. Evaluation of the Treatment of Controls over Efficiency and Effectiveness. 9. The rating system: The templates used a four-part rating system the language of which was built around the degree of certainty that could be assumed as to whether or not controls were working as intended. In making its ratings, IEG assembled a panel consisting 53 ANNEX E IEG COMPOSITE EVALUATION OF THE INTERNAL CONTROLS FRAMEWORK of its core consulting team and selected senior retired Bank staff. A summary of the four- part system and the language describing the ratings is in Box E.1. Box E.1. The Four-Part Rating System Rating Criterion Fully satisfactory High degree of certainty that the process element has been shown to conform to best practice, controls are well designed, operate Numerical equivalent: 1 effectively, etc. Satisfactory with qualifications Reasonable degree of certainty that process element has been shown to conform to best practice, controls are well designed, Numerical equivalent: 2 operate effectively; some qualifications/deficiencies. Moderately satisfactory Uncertainty, for any reason, that the process element has been shown to conform to best practice, controls are well designed, operate Numerical equivalent: 3 effectively, etc. or demonstrated significant deficiencies exist. Unsatisfactory Clear evidence that the process element has not been shown, or probably does not conform to best practice, is not well designed Numerical equivalent: 4 and/or operates ineffectively. Such cases would likely contain significant deficiencies that may contribute to a material weakness* Not applicable (NA) Not applicable * As defined by the Public Company Accounting Oversight Board, Audit Standards 2 and 5. Application of the Entity-Level Template: Summary of Results 10. Ratings of Approach and Method: As described above, the IEG template was used to evaluate four separate aspects of the overall review, the first of which dealt with management’s approach and method in completing its assessment of entity-level controls in Part II. The template contained two sets of questions in this segment, the first dealing with the design of assessment instruments (principally the ELCQ), the second with the use of data from the ELCQ and other sources. The results of the analysis are shown in Table E.1. IEG finds that the design of the assessment instruments was satisfactory, but with certain qualifications related mainly to the clarity of the ELCQ. Regarding the use of data from the ELCQ and other sources, IEG finds management’s approach fully satisfactory. Table E.1. Ratings from the Template IEG Ratings of Management’s Assessment Rating of Approach and Method Design of Assessment Instruments 1 Use of Data from ELCQ and Other Sources 1 OVERALL RATING: 1 11. The IAD Review and Opinion: In the case of the IAD Review and Opinion, the template had questions focused on four areas: Scope of Work, Criteria and Standards, Documentation and Evidence, and Quality of Conclusions. The questions were used as a 54 ANNEX E IEG COMPOSITE EVALUATION OF THE INTERNAL CONTROLS FRAMEWORK guide to evaluating the IAD review and opinion, but the applicability of the template was far less appropriate than in the case of its application to management’s assessment, because of the far fewer data points in the IAD review. Regarding the approach and method used in its Part II review, IAD was systematic in reviewing all management assessment materials and it made extensive use of ongoing IAD audits where relevant. IEG rated this approach fully satisfactory. With regard to the findings and the quality of conclusions, IEG identified similar weaknesses in controls, and notes that IAD found management’s qualified assurance to be fairly stated. However, IEG came to different conclusions as to the materiality of the weaknesses in controls over fraud and corruption. 12. Rating the Effectiveness of the Overall Controls Framework: In rating the effectiveness of the controls framework the IEG panel was guided by evidence from all sources. This included its review of all of management’s materials, the results from the ELCQ, and the IEG analysis conducted from the data revealed by the ELCQ responses, as well as other sources. It included consideration of the list of deficiencies IEG had compiled and the factors that combined to create a material weakness in IDA’s controls over fraud and corruption in its lending operations. As the template was organized around the five COSO components, following the way management had organized its assessment, the final ratings of controls effectiveness were also given for each component separately. They were then aggregated to give an overall rating for the whole framework. 13. As shown in Table E.2, IEG rated the effectiveness of controls in the framework as satisfactory with qualifications (rating 2), based on the material weakness and five significant deficiencies that had been identified. Also, since there were deficiencies in each of the COSO components, though they were rated separately, each component was also rated as having controls that were satisfactory with qualifications (rating 2). (It should be noted that controls effectiveness within each COSO component were evaluated and rated separately, based on the findings within each component. The fact that all ratings emerged as 2 is coincidental, but reflects deficiencies found in each component). Table E.2. IEG Rating of Overall Effectiveness of IDA’s Controls Framework COSO Components CE RA CA ML IC Effectiveness of Controls 2 2 2 2 2 Overall Rating 2 Satisfactory with Qualifications CE = Control Environment; RA = Risk Assessment; CA = Control Activities; ML= Monitoring and Learning; IC = Information and Communications 14. Rating Management’s Assessment of Controls over Efficiency and Effectiveness of IDA’s Operations: Management’s ratings of the controls over efficiency and effectiveness were built around the extent to which management had identified efficiency and effectiveness as key objectives in IDA operations, and then had also identified the full range of tracking systems and processes that the Bank has to keep management informed of the performance in each case. IEG noted that management did not give an account of actual performance of IDA in each area, but simply noted that controls were in place and were regularly operating in a way that ensured that Senior Management is well informed on a timely basis. IEG regards this as a necessary requirement and an acceptable approach to an 55 ANNEX E IEG COMPOSITE EVALUATION OF THE INTERNAL CONTROLS FRAMEWORK internal controls review. Accordingly, it rated management’s treatment as fully satisfactory in each case as summarized in Table E.3. However, IEG is also of the view that to give a sufficient account of controls over efficiency and effectiveness, at least some reference to indicators of efficiency and effectiveness outcomes is also relevant, and would be helpful. IEG made such references in its own evaluation. However, a full examination of efficiency and effectiveness outcomes, of a kind that IAD implied could have been attempted, is not, in IEG’s view, part of the scope of this review. Table E.3. Rating Management’s Assessment of Controls over Efficiency and Effectiveness of IDA’s Operations Controls over Effectiveness of IDA Operations 1 Effectiveness identified as key objective 1 Full range of effectiveness tracking systems identified 1 Controls over Efficiency of IDA Operations Efficiency identified as key objective 1 Full range of efficiency tracking systems identified 1 15. Behind the Overall Ratings: IEG has used exacting standards in its rating approach. This meant that in any given category, where one or more items may have received a rating less than 1 IEG tended to rate the overall item at less than 1, even though most items in a category may have been rated 1. This is evident in Figure E.2, which shows—for the rating of management’s assessment—data for the distribution of individual ratings across all categories. It is clear that ratings of 1 dominate. Out of 74 ratings, 43 (58 percent) were a rating of 1. Equally, more serious negative ratings (3 and 4) comprised no more than 10 percent of the total. Overall, therefore, the ratings of 2 (satisfactory with qualifications) that IEG gave to the all components tends to mask the frequent extent to which it also gave fully satisfactory ratings to many individual items and seriously negative ratings to only a few. Figure E.2. Distribution of Template Ratings 50 45 Number of responses 40 30 20 20 10 4 3 2 0 1 2 3 4 NA Response category 56 ANNEX E IEG COMPOSITE EVALUATION OF THE INTERNAL CONTROLS FRAMEWORK 16. Items Rated “Moderately Satisfactory” or Less: In eight cases, IEG gave individual ratings of moderately satisfactory or less to aspects of management’s assessment. The template questions for these items are shown in Box E.2. Most of these lower category ratings were given in relation to different aspects of risk assessment. Box E.2. Questions Which Led to IEG Ratings of Moderately Satisfactory or Less on COSO Components RISK ASSESSMENT To what extent did management address the issue of whether results-based vehicles (CASs, PADs, ISRs) reduced misallocation risks? Rating: 3 Rate the extent to which management’s assessment gave a fair and penetrating account of the effectiveness of both the design and effective operation of the Bank’s Integrated Risk Management Framework (IRMF). Rating: 3 Rate the extent to which management’s assessment showed how risk assessment mechanisms in the Bank and IDA differentiated between high level and major risk, and lower level and less serious risk factors. Rating: 4 Rate the extent to which management’s assessment systematically addressed the Risk Focal Points: Strategic Effectiveness; Operational Efficiency; Financial Soundness; Stakeholder Support. Rating: 4 CONTROL ACTIVITIES Where weaknesses were found, to what extent were these clearly distinguished between weaknesses in design (that is, flaws in the published procedures themselves) as against operation (which reflects noncompliance with established procedures), and were remedies suggested in each case that would be feasible to implement? Rating: 3 MONITORING AND LEARNING Rate the extent to which management’s review of the cluster of controls also touched on the monitoring of country outcomes, and IDA’s contribution to country outcomes, as undertaken to be done as part of the Monterey, Paris, and Marrakesh Consensus dealing with Results Measurement. Rating: 3 INFORMATION AND COMMUNICATION Rate the extent to which the management assessment threw light on whether information flows within the units are easy and open, and there are ways in which information critical to IDA controls can be communicated upstream within the organization. Rating: 4 57 Annex F: Statistical Appendix Table SA.1. Classification of Management's Questionnaire Responses (Based on Evaluated Pass Rates) Management Count by Two IEG Count by Four Categories Categories Distribution of Questions by Rating Category of Responses Yes, No, (Using IEG Yes No Total IEG Rating Yes No but but rating method) Rating Numbers 1 2 3 4 1 4 Control Environment 27 9 5 1 42 1.5 38 4 1.3 A. Integrity, ethics behavior 5 2 3 1 11 2 9 2 1.5 B. Control Consciousness, Style 11 3 14 1.2 13 1 1.2 C. Commitment to Competence 2 1 3 1.3 3 0 1.0 D. Org Structure, Auth, Respblty. 3 2 1 6 1.7 5 1 1.5 E. HR and Policies 6 1 1 8 1.4 8 0 1.0 Risk Assessment 22 11 1 1 35 1.5 35 0 1.0 Control Activities 11 7 2 20 1.6 20 0 1.0 Monitoring 26 4 30 1.1 30 0 1.0 Info. And Communications 17 10 1 2 30 1.6 27 3 1.3 Information 8 4 1 13 15 1 1.2 Communications 9 6 2 17 12 2 1.2 Totals And Overall Rating 103 41 9 4 157 1.5 150 7 1.1 Methodology: IEG classified the responses to the ELCQ according to its judgments on category of response. Overall ratings calculated using average category ratings weighted by numbers in each category Source: Management results matrix 59 ANNEX F STATISTICAL APPENDIX Table SA.2. The Distribution of "No" Responses by Frequency and COSO Components Number of "No" Number of "No" Number of "No" Totals % Distr. Questions Responses Questions Responses Questions Responses Frequency 1 to 2 3 to 5 >5 100 Control Environment 12 14 4 16 4 35 20 65 38 A. Integrity, ethics behavior 3 3 1 3 2 22 6 28 B. Control Consciousness, Style 5 6 1 3 0 0 6 9 C. Commitment to Competence 0 0 0 1 7 1 7 D. Org Structure, Auth, Respblty. 2 3 1 5 3 8 E. HR and Policies 2 2 1 5 1 6 4 13 Risk Assessment 16 23 2 8 0 0 18 31 15 Control Activities 12 17 3 9 0 0 15 26 16 Monitoring 2 2 0 0 0 0 2 2 1 Info. And Communications 8 13 2 9 3 31 15 53 29 Information 4 9 2 9 1 7 9 25 Communications 4 4 0 0 3 24 7 28 Totals 50 69 11 42 6 66 67 177 100 %Distribution 100 39 24 37 Source: Management results matrix. 60 ANNEX F STATISTICAL APPENDIX Table SA.3. Percentage Distribution of Questions by Category of Responses* Yes Yes, but Yes, but No No Total Overall 70 23 4 2 100 Control Environment 65 23 10 3 100 A. Integrity, ethics behavior 44 22 22 11 100 B. Control Consciousness, Style 79 21 0 0 100 C. Commitment to Competence 67 33 0 0 100 D. Org Structure, Auth, Respblty. 50 33 17 0 100 E. HR and Policies 75 13 13 0 100 Risk Assessment 76 24 0 0 100 Control Activities 58 37 5 0 100 Monitoring 87 13 100 Info. And Communications 63 26 4 7 100 Information 62 31 8 0 100 Communications 64 21 0 14 100 Methodology: IEG classified responses to the ELCQ according to its judgment on category of response Source: Management results matrix. *Data do not include questions on anti fraud programs and controls 61 ANNEX F STATISTICAL APPENDIX Table SA.4. Distribution of “No” Responses by COSO Components OVERALL Control Risk Control Monitoring and Information Environment Assessment Activities Learning and Communication 177 66 31 26 2 52 % Distribution (177=100) 37 18 15 1 29 3-5 “NO” RESPONSES PER QUESTION Number of Questions (8) 3 1 2 0 2 Number of “NOs” (33) 13 5 6 0 9 >5 “NO” RESPONSES PER QUESTION Number of Questions (6) 3 0 0 0 3 Number of “NOs” (52) 29 23 Source: Management results matrix. Data include "no" responses for Anti-Fraud Programs and Controls, within each COSO component 62 ANNEX F STATISTICAL APPENDIX Table SA.5. Negative Responses by Type of Controls Issue: Control Design or Control Operation Allocation by Category Percentages Design Operation Total Design Operation Control Environment I 24 17 41 59% 41% Risk Assessment II 20 9 29 69% 31% Control Activities III 12 8 20 60% 40% Monitoring and Learning IV 17 13 30 57% 43% Information and Communications V 16 11 27 59% 41% Fraud & Corruption VI 0 0 0 Totals 89 58 147 61% 39% Distribution of Ratings Design Operation Y Y/B Y/N N Y Y/B Y/N N Control Environment I 16 4 4 0 10 6 0 1 Risk Assessment II 13 7 0 0 9 0 0 0 Control Activities III 6 3 3 0 5 3 0 0 Monitoring and Learning IV 14 3 0 0 9 4 0 0 Information and Communications V 11 3 1 1 6 4 0 1 Fraud & Corruption VI 0 0 0 0 0 0 0 0 Totals 60 20 8 1 39 17 0 2 Percentage of Ratings by Category Design Operation Y Y/B Y/N N Y Y/B Y/N N Control Environment I 67% 17% 17% 0% 59% 35% 0% 6% Risk Assessment II 65% 35% 0% 0% 100% 0% 0% 0% Control Activities III 50% 25% 25% 0% 63% 38% 0% 0% Monitoring and Learning IV 82% 18% 0% 0% 69% 31% 0% 0% Information and Communications V 69% 19% 6% 6% 55% 36% 0% 9% Fraud & Corruption VI Totals 67% 22% 9% 1% 67% 29% 0% 3% Source:: IEG Calculations based on Management’s Results matrix 63 ANNEX F STATISTICAL APPENDIX Table SA.6. Portfolio Efficiency Indicators AFR SAR All Regions Lending Preparation Time (average months) /1 Q4FY04 17 13 16 Q4FY05 19 13 16 Q4FY06 14 17 15 Q4FY07 15 15 14 Q1FY08 17 24 17 Cost of Dropped Projects ($’000) /1 Q4FY04 8,523 1,339 26,130 Q4FY05 6,146 1,933 25,758 Q4FY06 10,569 1,519 23,047 Q4FY07 5,137 1,398 23,785 Q1FY08 7,116 1,936 29,692 Overage Projects (%) /1 Q4FY04 4 3.2 5.2 Q4FY05 3.1 3.1 5.2 Q4FY06 2.7 2 5.3 Q4FY07 3.1 1.2 4.5 Q1FY08 3 0.6 4.2 Completion Costs – Lending ($’000) /1 Q4FY04 665 313 462 Q4FY05 870 223 487 Q4FY06 655 261 430 Q4FY07 575 243 414 Q1FY08 568 237 412 Completion Costs – ESW ($’000) /1 Q4FY04 126 218 155 Q4FY05 157 220 178 Q4FY06 178 238 200 Q4FY07 190 263 212 Q1FY08 184 190 190 Completion Costs of TA /2 Q1FY05 137 90 85 Q1FY06 97 74 90 Q1FY07 140 183 114 Q1FY08 139 168 127 1/ 12 Months Ending Q4 FY04-07 2/ 12 Months Ending Q1 FY05-08 Source: Quarterly Reports to the Board, FY07 Quarter 4 and FY08 Quarter 1 64 ANNEX F STATISTICAL APPENDIX Table SA.7. Distribution of Entity Level Template Ratings Section 5 (Effectiveness of Controls) Section 4 Total (Approach Ratings Section Totals and Control Risk Control Monitoring Information and 5 Method) Environment Assessment Activities and Learning Communication 1 9 11 8 6 4 7 36 45 2 4 6 3 2 3 2 16 20 3 0 0 2 1 1 0 4 4 4 0 0 2 0 0 1 3 3 NA 0 2 0 0 0 0 2 2 65