EUROOPAN PARLAMENTTI Talousarvion valvontavaliokunta

Transkriptio

1 EUROOPAN PARLAMENTTI Talousarvion valvontavaliokunta TYÖASIAKIRJA Euroopan tilintarkastustuomioistuimen erityiskertomuksesta nro 18/2014 (vastuuvapaus 2014): EuropeAidin arviointijärjestelmät ja tulospainotteiset seurantajärjestelmät Talousarvion valvontavaliokunta Esittelijä: Bart Staes DT\ doc PE v02-00 Moninaisuudessaan yhtenäinen

2 Introduction In the current economic and political context, there is an increased need for the Commission to ensure and demonstrate that EU funds are used in an economic, efficient and effective manner. The European Parliament and the Council have indicated a need for a clearer view of the results achieved with regard to the EU s main objectives. Information on results achieved can only be obtained with a results accountability framework comprising three elements: monitoring, evaluation and reporting. Within the Commission s decentralised organisational framework, the Directorate General for Development and Cooperation - EuropeAid (EuropeAid) has set up its own results accountability framework which comprises the monitoring, evaluation and reporting of its activities. This report examines two of the key elements of this framework, namely evaluations and results oriented monitoring (ROM). Evaluation is the systematic and objective assessment of an ongoing or completed programme or policy, its design, implementation and results. Evaluation is essential for planning, designing and implementing EU policies and interventions, and enhancing transparency and democratic accountability. EuropeAid manages two types of evaluations, which are carried out by external contractors: (a) strategic evaluations, which comprise geographic evaluations (covering a country or region) and thematic evaluations, which cover a sector (e.g. education, food security) or a method of aid delivery (e.g. budget support, cooperation with development banks) and (b) programme evaluations, that can be carried out during the implementation (mid-term evaluation), on completion (final evaluation) or after completion (ex post evaluation) of an intervention. These evaluations assess performance against five criteria developed by the Development Assistance Committee of the Organisation for Economic Cooperation and Development (OECD-DAC), along with the international donor community relevance, effectiveness, efficiency, sustainability and impact. In addition, evaluations have to assess the EU s comparative advantage and the coherence with other EU policies and programmes, other donors actions and partner countries policies. ROM is a standardised external review, specific to external aid, designed to look at programmes performance through short on-site visits to programmes. The main purposes of these tools of the accountability framework are to improve the implementation of ongoing programmes and the design of future programmes and policies through feedback and lessons learned, and to provide a basis for accountability. Audit Scope and Objectives The Court examined whether EuropeAid has reliable evaluation and ROM systems. The audit focused on three main questions: - Are EuropeAid s evaluation and ROM functions appropriately organised and resourced? - Do EuropeAid s evaluation and ROM systems ensure the provision of relevant and robust findings? - Do the evaluation and ROM systems ensure the maximum use of the findings made? The audit covered the evaluation and ROM systems in place in the period from 2007 to PE v /6 DT\ doc

3 Where adequate information was available and could be checked by the Court, it took account of changes adopted or planned by EuropeAid for the period from 2014 to The audit was carried out between November 2013 and March 2014 and included (i) a review of EuropeAid s evaluation and ROM guidelines and of the main documents and processes in the management of evaluations, (ii) interviews with officials from eight units at EuropeAid, (iii) visits to the EU delegations in Benin and India, involving interviews with the delegations staff and a review of relevant documentation, (iv) a review of programming documents, programme financing agreements, strategic and programme evaluations, and ROM reports; and (v) a survey sent to 50 EU delegations to which 41 answers were received. Court's Findings and Observations Are EuropeAid s evaluation and ROM functions appropriately organised and resourced? The Court found that overall, EuropeAid s evaluation and ROM functions were well organised with a sound distribution of responsibilities between the Evaluation unit, the Quality and Results unit in charge of the ROM and the ROM coordinators but there was a lack of overall supervision by senior management of programme evaluation activities which are decentralised to EuropeAid operational units and EU delegations. Also, insufficient attention was paid to the efficient use of evaluation and ROM resources. EuropeAid has insufficient information on resources used by its evaluation and ROM systems. The total annual costs of evaluations and ROM would amount to million euro but EuropeAid has no reliable information on the number of programme evaluations and since their respective costs cannot be easily identified in the IT management information system. More EuropeAid had no management information on staff resources involved in ROM and programme evaluations; the Court estimated it to be likely above 80. Due to this lack of information, EuropeAid cannot ensure that staff and financial resources are appropriate and proportionately allocated to the various evaluation and ROM activities. Finally the Court observed that there was an inadequate coordination between programme evaluations and ROMs, which was likely to lead to inefficiencies. Programme evaluations and ROMs follow completely different planning approaches and criteria with the risk that certain programmes were over and under-assessed and sometimes not assessed at all. Do EuropeAid s evaluation and ROM systems ensure the provision of relevant and robust findings? The Court audit work revealed that the evaluation and ROM systems did not sufficiently ensure that relevant and robust findings are produced. If multiannual plans for strategic evaluations were based on appropriate criteria (achieving full geographic coverage, selection on the basis of the materiality funds allocated and increased number of evaluations of aid delivery methods or selection of thematic evaluations according to the priority themes of the Agenda for Change), the criteria for programme evaluation plans were based on insufficiently clear prioritisation criteria and significant differences in approach. The Court found that there was no monitoring system to identify and address frequent deviations from evaluation plans and to address the systemic causes of delay leading to a situation where only 70% of the selected evaluations were completed as planned. DT\ doc 3/6 PE v02-00

4 With regard to quality control procedures, the Court estimated that they were not implemented consistently for ROM and programme evaluations as reference groups (set up for each evaluation to advice all along the evaluation process) were not implemented for programme evaluations (only 39% of EU delegations set up a reference group always or most of the time and it was rarely the case by EuropeAid's operational units and EU delegations visited by the Court). More, the assessment of the final evaluation reports was only carried out by 71% of the EU delegations with only 37% of them filling assessment grids. With regard to the assessment of the quality of ROM reports, compulsory response sheets were only done for less than 60% of ROM reports from 2011 to Furthermore, the evaluation and ROM systems do not provide adequate information on results achieved due to insufficiently well-defined objectives and indicators, the latter being prepared in a political context and not sufficiently translated into operational objectives. The Court's sampling done for thematic, regional and country level programming documents and individual programmes also showed that objectives and indicators were unclear, lacking or without baselines or targets against which to assess performance. The absence of clearly defined objectives and indicators made difficult for an evaluation to establish evidence-based performance assessments. It also appeared from the Court audit work that evaluations and ROMs focused more on the implementation of programmes than on results achieved. There was a limited proportion of ex post evaluations and ex post ROMs for assessing impact and inherent limitations in the evaluation methodology for budget support i.e. difficulty to quantify the budget support contribution in a qualitative manner. Likewise, it should also be noted that the reporting from EU delegations did not provide much evidence on policy achievements but more on activities implemented and the EuropeAid's contribution to the Commission's Article 318 evaluation reports, despite positive developments, did not provide sufficient, relevant and reliable evidence on EU's policies achievement. The aforementioned factors limit considerably EuropeAid s capacity to account for the actual results achieved. Do the evaluation and ROM systems ensure the maximum use of the findings made? The Court found that the evaluation and ROM systems did not ensure that maximum use was made of findings because proper mechanisms were not in place to monitor their follow-up and dissemination. The Court identified weaknesses in the follow-up of strategic evaluations findings and recommendations issued, on their acceptance and use for new policies and programmes. Detailed information on the planned actions, their timetable and the allocation of responsibilities were also insufficient and the follow-up of the action plan by the Evaluation unit was limited to one year after the publication of the evaluation report. The follow-up for programme evaluations and ROM findings was not enough documented. The Court considered that in the absence of an appropriate follow-up system, EuropeAid could not ensure that findings and recommendations were used effectively. With regard to dissemination, it was found that evaluation and ROM findings were well disseminated to all EuropeAid s units, EU delegations, external stakeholders and other EU institutions and Member States. A synthesis of strategic evaluation results and key lessons learned and the aggregated ROM performance data were included annual activity report and in the preparation of the Commission s Article 318 evaluation report. However for PE v /6 DT\ doc

5 programme evaluations, the Court found that the dissemination was inadequate as the EAMRs reviewed by the Court contained no or little information in this respect. Moreover, programme evaluations report cannot be systematically used as there was no central database to manage, store and analyse ROM and programme evaluations. Esittelijän suositukset sisällytettäväksi mahdollisesti vuotuiseen mietintöön vastuuvapauden myöntämisestä komissiolle 1. suhtautuu myönteisesti erityiskertomukseen, jossa arvioidaan EuropeAidin arviointijärjestelmiä ja tulospainotteisia seurantajärjestelmiä, sekä esittää jäljempänä huomioitaan ja suosituksiaan; Yleisiä huomioita 2. on erittäin huolissaan EuropeAidin arviointijärjestelmien ja tulospainotteisten seurantajärjestelmien puutteellisesta luotettavuudesta, ohjelmien arvioinnin riittämättömästä valvonnasta ja seurannasta sekä myös EuropeAidin kyvyttömyydestä taata, että henkilöstöä ja rahavaroja kohdennetaan asianmukaisesti ja tehokkaasti eri arviointitoimiin; 3. huomauttaa, että parlamentille ja talousarvion valvontaviranomaiselle on ehdottomasti annettava selkeä kuva siitä, missä määrin unionin tärkeimmät tavoitteet on todella saavutettu; 4. muistuttaa, että Euroopan komission avustushankkeiden ja -ohjelmien saavutuksista olisi toimitettava ulkopuolista, objektiivista ja puolueetonta palautetta osana komission laadunvarmistusta koskevaa sitoumusta; 5. katsoo, että arviointien tulokset ovat keskeisiä panoksia, joita hyödyntämällä politiikassa ja politiikan tarkasteluprosessissa mukautetaan strategisia poliittisia tavoitteita ja parannetaan yleistä johdonmukaisuutta EU:n muiden politiikkojen kanssa; 6. uskoo, että panostamalla erityyppisten arviointien tuloksista saatavan näytön analysointiin ja yhdistämiseen saadaan kokonaiskuva suuntauksista mutta lisäksi voidaan tehdä päätelmiä, joilla vahvistetaan arviointimenettelyjen lopullista tehokkuutta ja tarjotaan samalla myös päätöksen- ja politiikantekoa varten parempaa näyttöä siitä, että tietyn avustusvälineen käytössä ollaan oikealla tiellä; 7. katsoo, että tiedon jakaminen kaikilla keinoilla ja välineillä on erittäin tärkeää kehitettäessä paitsi arviointikulttuuria myös erityisesti tehokasta suorituskulttuuria; 8. tukee tuomioistuimen suosituksia EuropeAidin arviointijärjestelmistä ja tulospainotteisista seurantajärjestelmistä. DT\ doc 5/6 PE v02-00

6 Tilintarkastustuomioistuimen suositukset Tuomioistuin esittää suosituksia arviointijärjestelmien ja tulospainotteisten seurantajärjestelmien tehokkaasta käytöstä, arviointien priorisoinnista ja seurannasta, laadunvalvontamenettelyjen täytäntöönpanosta, saatujen tulosten esittämisestä sekä arvioinnin ja tulospainotteisen seurannan tuloksia koskevista jatkotoimista ja tulosten levittämisestä. 9. EuropeAidin olisi pidettävä yllä riittäviä hallintatietoja ja suoritettava tarvearviointeja säännöllisesti varmistaakseen, että rahavaroja ja henkilöstöä kohdennetaan perustellusti ohjelmien arviointien ja tulospainotteisen seurannan välillä. 10. Varmistaakseen, että suoritetut arvioinnit heijastavat organisaation painopistealueita, EuropeAidin pitäisi määrittää selvät valintaperusteet ohjelma-arviointien priorisointia varten ja dokumentoida, miten perusteita on sovellettu arviointisuunnitelmien laatimisessa, sekä ottaa samalla huomioon täydentävyys tulospainotteisen seurannan kanssa; vahvistaa merkittävästi arviointisuunnitelmien täytäntöönpanon seuranta- ja raportointijärjestelmää sekä analysoida viivästysten syitä ja esittää kuvaus toimenpiteistä, joilla viivästyksiin on puututtu; vahvistaa EuropeAidin ohjelma-arviointitoimien yleistä valvontaa. 11. Ohjelma-arviointien ja tulospainotteisen seurannan laadun varmistamiseksi EuropeAidin pitäisi vaatia operatiivisia yksiköitä ja lähetystöjä noudattamaan laadunvalvontaa koskevia vaatimuksia sekä käyttämään ohjelmien arvioinneissa viiteryhmää ja suoritettujen laadunvalvontojen aineistoa; tarkistaa säännöllisesti, että kyseisiä valvontoja sovelletaan. 12. Parantaakseen arviointijärjestelmän valmiuksia toimittaa riittävästi tietoa saavutetuista tuloksista EuropeAidin pitäisi noudattaa tiukemmin määräyksiä, joissa vaaditaan käyttämään SMART-tavoitteita ja todennettavissa olevia indikaattoreita; muuttaa seurantajärjestelmää niin, että ohjelmista toimitettaisiin tietoja ainakin kolmen vuoden ajan niiden päättymisestä; ja lisätä merkittävästi ohjelmien jälkiarviointien osuutta. PE v /6 DT\ doc