The EUEuropean Union International Cooperation and Development Results Framework: A commentary
The European Commission has published what it calls ‘the EUEuropean Union International Cooperation and Development Results Framework’. This is a Staff Working Paper, but looks final rather than draft, and has been forwarded to Member States as such.
I am going to suggest further work, but let me say first that (a) it’s a good idea for an agency to have a results framework, (b) it’s a hard act to pull off, and (c) the Commission has made a creditable and credible stab at the exercise. Member States should certainly welcome the EUEuropean Union Results Framework. If they do not already have one of their own, they should start at once to prepare one.
There are, however, some issues on which it would be appropriate for Member States to ask for further work.
The outstanding items are:
- Thinking through the problem of tying the Results Framework to the SDGs, which are themselves illogical and overly complex.
- Broadening the scope. The results framework is presented as relating to ‘International Cooperation and Development’, which sounds all-encompassing, but in fact relates only to the work of the Development Commissioner, and within his remit, mainly to aid-financed activities. Thus, the Results Framework does not cover humanitarian aid, but more important has nothing yet to say about trade, climate change, foreign and security policy, migration, or any other aspects of the policy coherence agenda. Surely, Member States should ask that the Commission have a Results Framework covering all its activity? Perhaps the new Security Strategy will provide a template in that respect.
- The EUEuropean Union Framework is presented as ‘sectors’ rather than ‘goal areas’. This probably reflects the spending orientation of the document, but also seems to mix ends and means in the same way as the SDGs, and to be incomplete.
- When we come to the indicators, it is striking that both the Level 1 and Level 2 indicators are selective. In addition, some of the indicators are oddly placed; and some EUEuropean Union sectors have national indicators, but no indicators of EUEuropean Union performance. The main problem is that the EUEuropean Union has not articulated (in this document) a theory of change which links inputs to outputs to outcomes.
- Finally, it goes without saying that a final version of the Results Framework will need to list results, rather than just result areas or indicators.
What should happen next? A second edition, probably, with a revised series of high-level objectives, supported by a full theory of change to generate better indicators and areas of activity. In addition, the Commission and member States should also engage with Results Frameworks produced by developing countries, and should wherever possible align behind them. If everyone did this, the quality of joint programming would benefit hugely.
Why a Results Framework is useful
A results framework has obvious value, in a world in which the development sector has become preoccupied by impact. Just as an example, DFID published a paper on its results framework in September 2014. It summarises the purpose as follows:
‘DFID’s Results Framework is a tool that we will use to monitor and manage progress and report publicly on delivery. It sets out the development outcomes DFIDDepartment for International Development is seeking to contribute to, the actual results we will deliver, and the metrics we will use to measure our organisational effectiveness and efficiency.
This is the first results framework in which we have set out actual development outputs that DFIDDepartment for International Development will deliver against for which we can be held accountable. This marks a significant step forward. By measuring results we get a much better idea of what works and what does not so we can refine our programmes accordingly. We are also able to manage our resources to deliver these results.’
The DFIDDepartment for International Development Results Framework, applicable to bilateral aid only, is reproduced below. There is a separate table for the multilateral programme.
Figure 1
The DFIDDepartment for International Development Results Framework
Source: This all seems sensible, though some of the points I want to make about the EUEuropean Union Framework also apply to DFID. There are two generic issues that are perhaps worth pointing to straight away.
First, it is important for a Results Framework to be clear about the scope. In the case of the DFIDDepartment for International Development Framework, and also the EUEuropean Union Framework, as we shall see, the scope is limited to aid. This does not seem complete, in a world where development cooperation is increasingly seen to be about policy as well as aid – policy coherence, or Beyond Aid. The UK parliament’s International Development Select Committee criticised DFID for not reporting better on policy issues. It said that
‘We have made the point that action related to Policy Coherence for Development has not been well-reported in the past. There is a section in the latest Departmental Report which covers PCD, but the key themes are not visible in DIFD’s overall objectives or in its results frameworks, which focus too much on spending. Furthermore, DFIDDepartment for International Development only offers a partial lens on the UK Government’s policy and activities. A number of witnesses made the point to us that better reporting was required. We have demonstrated that many other countries offer comprehensive PCD reporting.’
In looking at solutions, the Committee received several practical suggestions that could be thought of as extended Results Frameworks. Thus,
‘A number of witnesses, including DFIDDepartment for International Development itself, noted the importance of cross-Government strategy documents (like the cross-Whitehall ‘Health is Global’ strategy), Cabinet Committees, jointly-owned cross-Government funds (like the International Climate Fund), joint ministerial appointments, and so on. There were many practical suggestions. For example, ODIOverseas Development Institute (London) proposed the “re-introduction of Public Service Agreements (or equivalent) (and) a reporting requirement to Parliament on PCD.” The British Overseas Aid Group made a different suggestion: the introduction of a “shared, coherent strategic policy framework to ensure government interventions across the board promote complementary policies.” UKAN interpreted this putative framework as a “Cross-Whitehall PCD strategy/action plan”, linked to a cross-departmental task force to track and monitor implementation.’
The second issue, of course, is that the whole results agenda is contested, or if not contested, then recognised to be replete with attribution and measurement problems. I have written about this before, most fully in 2011. Others also have much to say on the subject of reductionism, the bias to linear thinking, and the lack of attention to complexity and learning. See, for example, my review of Ben Ramalingam’s book on complexity, or Robert Chambers’ blistering attack on results-based aid. I was reminded the other day that Albert O. Hirschmann pioneered much of this thinking with his work on the ‘hiding hand’ in Development Projects Observed, first published in 1967.
Does this mean that results frameworks are useless? No, but they do need to be approached with common sense. As professor Adrian Wood observed about the MDGs, ‘take them seriously, but not literally’. I wrote back in 2011 about what I called a post-Fordist approach to results, or Results 2.0, concluding that it was important to avoid ‘bean-counting’ (Andrew Mitchell’s phrase), and be sensitive to qualitative measures, as well as indirect effects.
Introducing the Results Framework
Let’s turn to the EUEuropean Union initiative. The Staff Working Paper explains the purpose of the exercise, contextualises the framework in relation to the ongoing SDG discussions, and summarises how data will be collected and used. The actual framework is in a series of appendices.
The overall structure is as in Figure 2 below. There are three levels, dealing respectively with development progress in EUEuropean Union partner countries, EUEuropean Union contributions to the above, and organisational performance. These notes concern only Levels 1 and 2.
Figure 2
Source: http://data.consilium.europa.eu/doc/document/ST-7604-2015-INIT/en/pdf
For ease of reference, I have reorganised the material in the appendices so as to correspond to the 17 proposed SDGs.
The EUEuropean Union has 15 ‘sectors’, as in Table 1 below. These can be compared to the proposed SDGs, as in Table 2. The SDG links are as given by the Commission. It can immediately be seen that some SDGs have no corresponding EUEuropean Union sector, though in fact some of the indicators (see below) do in fact relate to ‘missing SDGs’. Water and sanitation is one example, inequality another, climate change a third.
Table 1
Source: http://data.consilium.europa.eu/doc/document/ST-7604-2015-INIT/en/pdf
Table 2
Source: http://data.consilium.europa.eu/doc/document/ST-7604-2015-INIT/en/pdf
When it comes to indicators, the details are given in Table 3.
Table 3
Source: http://data.consilium.europa.eu/doc/document/ST-7604-2015-INIT/en/pdf
Discussion
First, it is worth reminding ourselves that the SDGs are work in progress and may change by September 2015, when they are to be agreed in New York. Actually, this looks increasingly unlikely, but I for one will be pleased if they do – see earlier pieces here and here for the reasons why. Among the many reasons to be irritated by the current draft is the failure to distinguish properly between ends and means. For example, growth is not an end in itself, rather a means to other ends, like jobs, income and security. Indeed, jobs are also a means to an end. The structure of the SDGs reflects a lack of literacy as to how logical frameworks and results matrices are supposed to work. A question, then for the EU: is it sensible to tie the results framework to a patently inadequate set of SDGs?
Second, the results framework is presented as relating to ‘International Cooperation and Development’, which sounds all-encompassing, but in fact relates only to the work of the Development Commissioner, and within his remit, mainly to aid-financed activities. It just happens that the relevant service was renamed at the beginning of the Juncker Commission, from ‘Development’ to ‘International Cooperation and Development’. Thus, the Results Framework does not cover humanitarian aid, but more important has nothing to say about trade, climate change, foreign and security policy, migration, or any other aspects of the policy coherence agenda. This is surely a problem. Should not Member States insist that the Commission have a Results Framework covering all its activity? Perhaps the new Security Strategy will provide a template in that respect.
Third, it is interesting that the EUEuropean Union Framework is presented as ‘sectors’ rather than ‘goal areas’. This probably reflects the spending bias of the document, but also seems to mix ends and means in the same way as the SDGs, and to be incomplete. On the latter point, why, for example, is nutrition defined as agriculture only, and not health (and we will come shortly to the importance of transitory as well as chronic food insecurity)? And why are food shocks the only kinds of shocks considered? Don’t financial or commodity price shocks count? Or extreme weather events?
Fourth, we then come to the indicators, where it is striking that both the Level 1 and Level 2 indicators are incomplete. In addition, some of the indicators are oddly placed; and some EUEuropean Union sectors have national indicators, but no indicators of EUEuropean Union performance. For example (and selectively, because similar points could be made about most of the sectors):
- The poverty ‘sector’ has a poverty headcount indicator, but not a poverty gap indicator, and has no EU-relevant targets at all. There are no non-income measures of poverty included. There is an inequality target which does not belong here, and also has no EU-relevant targets.
- The nutrition indicator is for stunting, with no indicator for wasting. The EU-level indicator is for only one possible ‘driver’ of changes in nutrition status, viz nutrition programmes, with no recognition of other drivers, like growth, health care, education programmes and so on.
- The Sustainable and Inclusive Agriculture sector has only one national-level indicator, added-value per hectare, which may (but may not) measure productivity increases, but certainly does not measure either sustainability or inclusiveness. The EU-level indicators are partial.
- The employment and social protection sector has two Level 1 indicators, viz the proportion of employed people living below $US1/day and the share of older people receiving pensions. That is very limited on both counts. Where, for example, is the number of ’jobs’ or livelihoods, including those in the informal sector? The Level 2 indicator, there is only one, if for the number of people who have benefited from EU-supported active labour market programmes, so no mention of social protection.
I could go on in this vein. The main problem is that the EUEuropean Union has not articulated (in this document) a theory of change which links inputs to outputs to outcomes.
Finally, it goes without saying that a Results Framework needs to list results, rather than just result areas or indicators. Presumably this is the next step.
What would be better?
What would be better? I started out by saying that this exercise was not straightforward. At the same time, the EUEuropean Union would be well-served by further work on the high-level objectives, supported by a full theory of change to generate better indicators and areas of activity.
Member States should certainly welcome the EUEuropean Union Results Framework. If they do not already have one of their own, they should start at once to prepare one. They should also engage with Results Frameworks produced by developing countries, and should wherever possible align behind them. If everyone did this, the quality of joint programming would benefit hugely.
Read more...