What is the ICAI up to, what have we learned, and what should happen next?

These are exciting times in the world of independent aid evaluation – at least in the UK. The Independent Commission on Aid Impact has completed its second year of work and published its latest Annual Report . Meanwhile, the Cabinet Office is conducting a review of ICAI, oddly billed as a triennial review . And the International Development Select Committee, to which ICAI reports, is collecting written evidence and holding a public oral session on the lessons learned from ICAI's work, as well as on its future role. It is worth noting that the IDCInternational Development Committee has established a special sub-committee to manage ICAI, chaired by Richard Burden MP. Oh, and the Shadow Secretary of State, Ivan Lewis, announced back in January that under a future Labour Government, 'the independent commission on aid effectiveness will have greater independence from ministers and have wider powers to report on all aspects of UK development policy'. So, what is ICAI up to, what have we learned, and what should happen next?

I addressed all these questions in some detail a year ago, and was fairly sympathetic to the enterprise. That was not the case for all who commented on my piece, including some who have submitted written evidence to the IDCInternational Development Committee this time round. Watch out for sharp debate about ICAI's mandate and work programme.

ICAI's mandate

Let's start with the mandate. ICAI itself describes itself as 'the independent bodyresponsible for the scrutiny of UK aid. ICAI focuses on maximising the impact and effectiveness of the UK aid budget for intended beneficiaries and the delivery of value for money for the UK taxpayer. . . (It) is a unique body that publishes transparent, impartial and objective reports on the effectiveness of UK aid'. That sounds fairly straightforward. The debate has been about whether ICAI is comprehensive in its coverage (it is not), and about how it complements other internal and external evaluation by DFID. On the latter question, DIFD's Permanent Secretary, Mark Lowcock, was unambiguous.  Speaking to the Select Committee in December 2011 he said that 'on the evaluation department, that used to be the bit of DFIDfrom which we ran our programme of independent evaluation studies. At the time ICAI was established, we closed that business down. We do not ourselves, from the central department, produce those internal independent evaluations any more. We have recycled the money from that operation into other things'.

If that is the case, it is rather surprising to visit the DFIDDepartment for International Development website and find quite a list of recent evaluation reports , several (but I think not all) conducted jointly with other donors. It is also interesting to discover that DFIDDepartment for International Development has awarded a Global Evaluation Framework Agreement , providing for £150m worth of evaluation work over two years, to be delivered through a consortium of organisations, and designed 'to ensure the provision of efficient and effective consultancy services for the design and implementation of evaluations across its programmes, . . . (and to) become DFID's primary route for engaging service providers to undertake its evaluations'. By contrast, ICAI's expenditure in 2012-13 was £2.9m .

Perhaps the answer is to be found in a new Evaluation Policy , published in May 2013. In his Preface, Mark Lowcock states that the creation of ICAI 'has been matched by the embedding of evaluation across DFIDDepartment for International Development to ensure that lessons are learned during the development process'. I wonder what 'embedding' means? Specifically, where do the boundaries lie of ICAI's remit? Presumably the 'non-embedded'? Or is the distinguishing factor independence, which implies that DFID's own evaluations are not. And where does that leave the work of the National Audit Office?

Researching the topic, I found the following list of '15 types of evaluation', presented by DFIDDepartment for International Development at a meeting on GEFA of the types of work it would like its contractors to undertake. Note that the word 'embedded' does not appear in the list. Nor does the word 'independent'. Nor, for that matter, do the words 'impact' or 'value for money', both central to ICAI's work. But I wonder whether ICAI thinks it should be responsible for thematic or sectoral evaluations, or indeed summative or meta-evaluations: all of those would seem to fit within its remit, and, indeed, play to its comparative advantage.

• Quantitative
• Qualitative
• Mixed Methods
• Experimental/ Quasi Experimental
• Formulation of evaluation questions
• Policy
• Thematic/ Sectoral
• Process
• Country Programme
• Formative
• Joint
• Looking at Theory of Change
• Summative
• Meta
• Participatory Evaluation

I am sure there is no sleight of hand intended here, but clarification would be useful. There also needs to be coordination of reporting. Mark Lowcock says in his Preface to the Evaluation Policy that 'progress will be captured in the Annual Evaluation Report'. That's good. How will ICAI's Annual Report sit with this?

ICAI's work programme

Mandate aside, and possible overlap with DFID's evaluation efforts, is ICAI covering the ground? We know it does not set out to be comprehensive (see last year's debate on that question, and evidence submitted to this year's enquiry) – unlike the Independent Evaluation Group at the World Bank - but does its selection of reports offer an adequate purposive sample of HMG's work on international development (NB not just DFID's)? Last year, there were eleven reports. This year, ICAI has published a further twelve. The work programme for 2013-14 will see a further twelve. A complete list of these 35 reports is appended.

There is quite a range in this set. Analysis of methodologies. Work on a dozen countries and several regional programmes. Engagement with seven multilateral programmes. Half a dozen thematic evaluations. Some cross-Government work. And a couple of high profile reviews of DFID's use of contractors and relationships with NGOs. Not all these have yet delivered, but Year 3 is presumably in train. And, actually, that's a good list and range. There have been criticisms that ICAI is too focused on fiduciary matters, and that it behaves like an auditor rather than an evaluator. I am inclined to agree with both those points. I have also argued that ICAI should do more on policy. However, to use a golfing metaphor, its initial shot selection seems fine. If there is a problem, it might lie more with execution: time for ICAI to work on its swing?

What would be interesting would be to map ICAI's work systematically across DFID's programmes, and against the 15 evaluation types reproduced above. The Select Committee might ask ICAI to do this. It would also be good to factor the NAO and the IDC's own work programme into the mix. Are there any obvious gaps?

What have we learned?

It goes without saying that the two dozen reports already published provide a great deal of information and analysis. I observed last year that ICAI has produced informative and readable reports, and that remains true. The scoring in individual reports is broadly comparable to last year, which is to say mostly OK, with few outstanding scores and few disastrous one. There is one overall 'green' rating, for the livelihoods programme in Western Odisha in India. There are no overall reds, though four 'amber-reds', including for the EU's work in Low Income Countries and for the Conflict Pool.

The big challenge is to generalise from the reports and construct an overall narrative about UK development cooperation. Last year, I observed that ICAI itself had abdicated responsibility in this regard, but that I had managed to identify five cross-cutting issues, viz

1. What is 'impact'?
2. The idea of buying a seat at the table
3. The staffing and skills needed to deliver high quality programmes
4. Bilateral versus multilateral aid
5. DFID as a venture capitalist

I wish that ICAI had taken these up itself in this year's annual report. Instead, it suggests a few 'key themes' of its own, as in the list below – and with the full text reproduced in the annex to this note. There is some overlap, but I find ICAI's key themes to be somewhat unadventurous and cautiously written. Anyway, their list is:

• The value of designing a clear 'theory of change';
• The need for a differentiated engagement strategy with multilaterals;
• The need for DFIDDepartment for International Development to pay more attention to the skills needed for delivery, including in managing contractors and multilaterals, and to address a possible trade-off between efficiency and lengthening delivery channels.;
• A need better to understand the reasons for and variations in administrative overheads as between agencies;
• The value of beneficiary involvement in programme design and execution; and
• Building the capacity to learn and change course as a result of learning.

The Commissioners also observe, echoing one of my points last year, that 'we have been struck by the number of internal and external discussions that we have had about the concept of impact, what it means and how to measure it in an international development context'. They are planning a study on this, but not until year 4 (2014-15). Why wait?

This year's reports do not map perfectly onto themes carried over from last year. For example, there is little analysis this year of what it takes to 'buy a seat at the table'. However, there is some overlap with previous work, and with ICAI's own list, in the four themes I pulled out of this year's work:

1.  First, how well does DFIDDepartment for International Development work with multilaterals? There is evidence in the ADB, Horn humanitarian, EUEuropean Union and UNICEFUnited Nations Children's Fund reports.
2.  Second, how good is cross-Government cooperation, esp but not only on 'political' or security issues. There is evidence in the Conflict Pool and Arab Spring reports, but also in the Nepal report.
3.  Third, how well does DFIDDepartment for International Development manage outside intermediaries apart from the multilaterals: evidence in the PPAPartnership Programme Agreement and Contractor reports.
4.  Fourth, how well is DFIDDepartment for International Development conforming to aid effectiveness principles post-Busan: evidence in the Pakistan, Nigeria, Sudan and West Odisha reports.

(a) Working with multilaterals

There are three reports this year which deal specifically with aid spending through multilaterals, namely those covering UNICEF, the Asian Development Bank and the EU. In addition, other reports touch on multilateral issues, for example in relation to humanitarian aid in the Horn of Africa. The multilateral issue is of intrinsic interest, but is also timely, because DFIDDepartment for International Development is currently revising the Multilateral Aid Review it first published in 2011. The Select Committee is reviewing the MAR, considering multilateral aid in the context of its enquiry into the future of UK development cooperation , and also thinking about the desirability of setting up a UK development finance institution , to offer loans: both the EUEuropean Union and the ADBAsian Development Bank have relevant experience here.

There is a case for saying that multilateral aid should have figured more prominently in ICAI's work. The latest OECD/DAC report on multilateral aid , for 2012, shows that the UK is one of the heaviest users of the system, with over half of total oda channelled through multilaterals in 2010: the UK is behind only Italy and Luxembourg in this respect.

Interestingly, nearly 40% of the funds channelled multilaterally are additional to normal core contributions, so DFIDDepartment for International Development must see special advantages in using the multilateral system, especially the UN and the World Bank. This is the kind of meta-evaluation question it would be useful for ICAI to explore. This year, their main comment is that 'multilateral expenditure frequently offers an opportunity for DFIDDepartment for International Development to deliver at greater scale and in sectors where it does not usually operate, such as road-building and infrastructure'. A more sophisticated analysis would be useful, discriminating between multilaterals. ICAI could then contribute more useful to wider reviews, like DIFD's own MAR, and joint donor exercises like MOPAN. It would also be useful to ask routinely whether bilateral programmes could have been delivered more effectively by multilaterals – and perhaps vice-versa.

The ADB report is useful in identifying three roles for DFID, as shareholder, contributor and co-financer. This is a distinction that could be applied to all ICAI's analysis of multilateral aid – and indeed by DFID. Will it be taken up, I wonder, by the MAR review? I wonder also whether ICAI considered using this approach for its review of UNICEFUnited Nations Children's Fund and the EU? The shareholder optic might be especially useful in assessing DFID's role in shaping policy. There is not much on this in any of the three reports – most especially the EU, where it is a big part of the job.

None of DFID's roles is easy to fulfil. There are big differences between the reports in DFID's level of engagement – from intensive (ADB?) to 'light touch' (UNICEF). On some occasions, DFIDDepartment for International Development play a leadership role in donor relationships and coordination (as in the Horn), but this appears to be variable (e.g. complaints about not doing enough on the ground wrt ADBAsian Development Bank and EU). Why? And what is it reasonable to expect? ICAI note this, observing that 'a key question we have grappled with in our reports in this area has revolved around the appropriate level of DFIDDepartment for International Development oversight required to assess impact'. The words 'required to assess impact' could easily be omitted to make the observation more general. It is a pity that ICAI did not venture an opinion on what might be the criteria for deciding how much engagement is appropriate and when (in terms of agency funding cycles, for example). For example, surely DFIDDepartment for International Development should not be in the business of micro-managing multilaterals.

Finally, it is just worth noting that although ICAI is only mildly enthusiastic about the Asian Development Bank, scoring it Green-Amber, there may well be lessons to learn about how to manage a loan portfolio, and how to combine loans and grants. Further, there are important questions, not dealt with in the ICAI report, concerning the comparative advantage of using the ADBAsian Development Bank as a vehicle in Asia, as opposed to creating a new UK development finance institution. Or, for that matter, the IDAInternational Development Association (of World Bank) or IFC, both active in the region.

(b) Cross-Government cooperation

Cross-Government working is a recurrent theme of those of us who write about the 'new development' agenda of global public goods and global deal-making, rather than only the 'old aid' agenda of public service delivery: about climate change, security and global financial stability, as well as about maternal mortality and child malnutrition. This is also a preoccupation of those concerned with Policy Coherence for Development; and indeed of the Select Committee, in its enquiry into the future of UK development cooperation . Cross-Government working is obviously needed, and obviously has development implications: but does it work?

We will learn more from ICAI's forthcoming reports, including the International Climate Fund and trade in Southern Africa (provided they ask the right questions and do not just focus on DFIDDepartment for International Development 'projects'). Meanwhile, this year, there is important evidence in the reports on the Conflict Pool and, though published after the end of the year, on the Arab Spring.

It is fair to say that ICAI is sceptical about the Conflict Pool, an inter-departmental instrument shared by DFID, the FCOForeign and Commonwealth Office and MOD. It scores amber-red, with ICAI concluding that

'The Conflict Pool was established to combine the skills of the three departments in defence, diplomacy and development into a coherent UK approach to conflict prevention. It has proved effective at identifying and supporting worthwhile conflict prevention initiatives and has delivered some useful, if localised, results. It has, nonetheless, struggled to demonstrate strategic impact: it lacks a clear strategic framework and robust funding model; its governance and management arrangements are cumbersome; and it has little capacity for measuring results. While, in the past, the Conflict Pool may have contributed to a more joined-up approach to conflict prevention, the inter-departmental co-ordination role is now played more effectively by other mechanisms. The task of administering the Conflict Pool by consensus across three departments is so challenging that those charged with its management have tended to shy away from harder strategic issues.'

This finding is in marked contrast to the National Audit Office, which also reviewed the Conflict Pool in 2012, and concluded that

'The tri-departmental nature of the Pool works well. There are clear high level common objectives for all parties and the structure promotes joint working at the top; the Board and Secretariat communicate well and enjoy a positive relationship. At a lower level, we found departmental teams have their own aims and approaches to conflict prevention, but this can provide positive challenge to projects, for example, at Programme Boards. Delivery overseas is largely FCO, DFIDDepartment for International Development or MODMinistry of Defence specific, depending on the required intervention, though there is a desire among project teams we interviewed to work more closely with colleagues from the other departments. . .  The good practice evidenced here in working collaboratively should be shared with the rest of Whitehall. More joint working in-country could be encouraged and facilitated by the Centre, for example by setting joint indicators or using staff more flexibly between departments.'

It would be useful for this difference of view to be probed in more detail. It is also interesting to ask why it is that the NAO and ICAI carried out similar enquires at the same time.

The ICAI judgement on FCOForeign and Commonwealth Office and British Council actions in response to the Arab Spring is more positive, scoring green-amber. Interestingly, DFIDDepartment for International Development is very much involved. ICAI concludes that

'the Arab Partnership represents a strong model of inter-departmental coordination. Within the overall strategy, there is a clear division of labour between the FCOForeign and Commonwealth Office and DFID, based on individual comparative advantage. We were impressed by the way in which DFIDDepartment for International Development resources have been used to boost FCOForeign and Commonwealth Office delivery capacity. This has included DFID's contribution of £20 million to APPF, staff secondments and transfers and the provision of advisory staff as needed. . . These collaborative structures are the more impressive considering that they were created in a situation of considerable urgency. The two departments have shown a commendable willingness to innovate and to adjust their ways of working as necessary'.

There seem to be several elements conducive to success, apart from goodwill and flexibility: a clear cross-Government strategy, an effective inter-Departmental Board, a dedicated Secretariat, a series of Programme Boards on different topics and specialised in-country teams. It would have been useful for ICAI to draw these out, and then to test them in its forthcoming enquiries, including on climate finance.

Some further questions also follow:

1. It would be interesting to ask what the difference is between a joined-up pool and a forum or mechanism for coordination between ministries? And when is one preferable to the other?
2. How does ministerial leadership and accountability work when cross-cutting programmes are established? Does it work better when the National Security Council is involved? Are Cabinet committees the right mechanism? Or is there some advantage in having a shared minister (as has occurred when DFIDDepartment for International Development and the then DTIDepartment of Trade and Industry shared a trade minister)?
3. Is there a reason why DFIDDepartment for International Development cannot just implement all programmes – it has much better capacity and experience than the FCOForeign and Commonwealth Office and in some ways the British Council. Would this not guarantee better joined-up thinking?
4. Are there other cases where it would be useful to consider a joint approach – for example, Nepal, where DFIDDepartment for International Development seems to have acted independently.
5. Finally, one is left asking what the case is for cross-government initiatives. If they are important on e.g. conflict, why not also trade, migration, technology, finance, and many other issues.

(c) Managing outside intermediaries

This is a theme that ICAI itself has picked up as a cross-cutting issue – perhaps not surprisingly, given its high profile reports on both NGONon-governmental organisation partnership agreements and outside contractors. Their comments in the Annual Report are brief, however. They say merely that 'our report on DFID's use of contractors noted some shortcomings in programme management capability and staff continuity which led to poor mobilisation of contractor-run projects'.

The contractor report is more pointed. It says that 'DFID's poor end-to-end programme management, however, has led to delays. Learning is not captured from contractors and used to inform future programming. The reform of PrG has improved processes but is too slow and lacks prioritisation. As a result, decisions to use contractors are not guided by a strategic plan to deploy the right contractors, including major, niche and innovative new entrant organisations, to best effect'.

The PPAPartnership Programme Agreement Report says that 'DFID would achieve more with its PPAs if it were to refocus on the added value they can provide as a strategic instrument, in particular when contrasted with the other CSOCivil Society Organisation funding mechanisms that DFIDDepartment for International Development uses'.

There are a couple of issues here. First, it would be helpful to recognise that DFIDDepartment for International Development is mainly a wholesaler rather than a retailer in delivering aid, and necessarily works through intermediaries. This is not the case for budget support, and perhaps not for some projects, but generally DFIDDepartment for International Development depends on others. This was a point made strongly in the Humanitarian Emergency Relief Review, for example: DFIDDepartment for International Development needed core skills in managing its intermediaries. It would be interesting for ICAI to explore this issue and examine the question of which kinds of projects, in which places, and on which time-scales, benefit most from particular kinds of intermediary. Perhaps this is what ICAI mean by saying that 'DFID faces real challenges and trade-offs in designing delivery channels of appropriate length'. Does it have answers?

A more serious question is about the extent to which DFIDDepartment for International Development sees its NGONon-governmental organisation and private sector contractors as agents or partners. ICAI seems clearly to think that DFIDDepartment for International Development sees both NGOs and contractors as basically agents – hence the idea of 'refocusing' PPAs on the value-added they can provide to DFID, and the emphasis in the contractor report on service delivery.

For PPAs, it ought to be axiomatic that they are a partnership between DFIDDepartment for International Development and the NGO, the result of a negotiation about means and ends. Hint: the clue is in the name, Programme Partnership Agreements. An important issue to investigate, therefore, is the history and quality of the partnership. Here, ICAI is somewhat ambivalent. On the one hand, it concludes that ' DFIDDepartment for International Development would achieve more with its PPAs if it were to refocus on the added value they can provide as a strategic instrument, in particular when contrasted with the other CSOCivil Society Organisation funding mechanisms that DFIDDepartment for International Development uses'. On the other hand it observes that 'DFID has prioritised accountability over partnership' and says that 'We are concerned that the PPAs have not, in practice, operated as partnerships. The CSOs were selected on the basis that they share objectives with DFID. They have knowledge, influence and expertise that could contribute to DFID's work. DFIDDepartment for International Development failed, however, to define what it hoped to gain from these partnerships and, as a result, has gained less than it might have done'. The loss probably works both ways. ICAI recommend appointing a substantive expert to work with each PPAPartnership Programme Agreement holder, but this might not be enough to bring about the culture change required throughout the organisation.

When it comes to consultants, the language is decidedly less empathetic, reflecting an emphasis on service delivery and performance management. This can be seen clearly in DFID's 'Statement of priorities and expectations for suppliers ', published in January 2013. This manages to list 20 demands of contractors, under such headings as 'value for money' and 'aligning with DFID', without once suggesting anything DFIDDepartment for International Development might offer in return: for example, 'simplify procedures' or 'reduce reporting requirements'. ICAI is probably right to suggest that a new approach is needed, perhaps more of a partnership.

(d) ICAI's Pakistan, Nigeria, Sudan and West Odisha reports all raise issues about aid effectiveness. Each contains a wealth of information. It would be interesting to ask to summarise what they see as the key elements of the evolving Rome/Paris/Accra/Busan consensus on aid effectiveness, including but not just fragile states, and say what they think DFID's (or HMG's) main strengths and weaknesses are with respect to the aid affectiveness agenda.

The core principles of aid effectiveness have remained constant since the Paris Declaration was agreed in 2005: ownership, alignment, harmonisation, results, and mutual accountability. The Paris Agreement was updated in Accra in 2008, and again in Busan in 2011.

I wrote about Busan at the time (see here ), identifying eight innovations, as follows:

1. 'The full engagement of non- DAC donors, especially emerging economies;
2. Reflecting this, the emphasis on South-South cooperation and on differentiated responsibilities of new and traditional donors;
3. The involvement as full participants of non-official actors, especially civil society;
4. Recognition that development cooperation goes beyond aid, to include other flows, technology transfer etc . . .
5. The commitment to transparency;
6. The compact on fragile states;
7. A commitment to use country systems as the default, and requiring donor countries to explain formally why they do not do so; and
8. Specific dates for a number of commitments, mainly to processes rather than specific outcomes: to review plans for untying aid in 2012; to publish a standard for aid transparency by December 2012, along with an implementation schedule; by 2013, to provide recipient countries with rolling 3-5 year indicative forward expenditure plans; also by 2013, to make greater use of country-led coordination arrangements and division of labour; agree on principles to reduce proliferation of channels by the end of 2012; agree principles to deal with aid orphans, by 2012; agree new indicators for monitoring progress, by June 2012.

One other major point is the creation of a new Global Partnership for Effective Development Cooperation, to replace the Working Party on Aid Effectiveness, and, though this is not quite stated explicitly, to encompass the work of the UN's Development Cooperation Forum : at least the UN is mandated to work with the DAC on the new Partnership'.

The UK, of course, signed up to these commitments. It is rather surprising that ICAI does not routinely assess, in its individual reports, but especially in its annual report, progress towards the various targets. It would be useful if the Commissioners at least expressed a view.

What should happen next?

This is a question that will shortly receive one answer provided by the Cabinet Office Review. My own view is that ICAI has proved its usefulness, but (a) needs to delineate better its division of labour with the NAO and DFID's own evaluation, and (b) needs to venture more bravely into the high-level policy debates that will shape the future of DFID. I have just helped to run a workshop at ODIOverseas Development Institute (London) on the future of development cooperation, and have reported elsewhere the comment of one donor representative that 'we are terrified by the extent and speed of change that will be needed in our institution' to ensure future relevance. Just say that that applied to DFID. Could we rely on ICAI to have the answers? Not yet. But we should.

 Note: a versoin of this was submitted to the International Development Select Committee, to which I am a paid specialist adviser.

Annex 1

List of ICAI Reports

Year 1

1. ICAI's Approach to Effectiveness and Value for Money
2. DFID's Approach to Anti-Corruption
3. DFID's Climate Change Programme in Bangladesh
4. DFID's Support to the Health Sector in Zimbabwe
5. The effectiveness of DFID's Engagement with the World Bank
6. DFID Programme Controls and Assurance in Afghanistan
7. Girl Hub: a DFIDDepartment for International Development and Nike Foundation Initiative
8. Evaluation of DFID's Electoral Support Through UNDP
9. The Management of UK Budget Support Operations
10. Evaluation of DFID's Support for Health and Education in India
11. DFID's Education Programmes in Three East African Countries

Year 2

1. Evaluation of the Inter-Departmental Conflict Pool
2. The Effectiveness of DFID's Engagement with the Asian Development Bank
3. DFID's Humanitarian Emergency Response in the Horn of Africa
4. Evaluation of DFID's Bilateral Aid to Pakistan
5. DFID's Education Programmes in Nigeria
6. DFID's Oversight of the EU's Aid to Low-Income Countries
7. DFID's Livelihoods Work in Western Odisha
8. DFID's Water, Sanitation and Hygiene Programming in Sudan
9. DFID's Peace and Security Programme in Nepal
10. DFID's work through UNICEF
11. DFID's Use of Contractors
12. DFID's Support for Civil Society Organisations through Programme Partnership Arrangements

Year 3

1. DFID's health programmes in Burma
2. FCO and British Council Aid Response to the Arab Spring
3. DFID's support to Montserrat
4. DFID's trade development work in Southern Africa
5. DFID's Support to Refugees through UNRWA
6. DFID's approach to empowerment and accountability
7. DFID's support to agricultural research
8. Learning within DFID
9. DFID's Afghanistan country programme
10. DFID's work with the private sector
11. DFID's support to GAVI
12. International Climate Fund

Source : http://icai.independent.gov.uk/

Annex 2

Extract from ICAI's Annual Report

Key Themes

5. In this section we set out some of the key themes we have observed about ODAOverseas Development Assistance expenditure, under each of our four guiding criteria. Our commentary relates to DFIDDepartment for International Development unless specified otherwise.

Objectives

6. Theories of change: our reports show that DFIDDepartment for International Development and other departments can usually demonstrate a clear link between their strategic objectives and the programmes they establish. The next challenge is planning programmes so as to deliver the desired outcomes – the so-called 'theory of change'. Getting this right is critical to the ensuing success of each programme and we have seen both good and bad examples of this in our work this year.

7. Setting out a clear theory of change is clearly easier in some situations than others. For example, where DFIDDepartment for International Development spends money through multilateral organisations or on programmes designed to stimulate changes in the political or business environment, there are many more external factors, uncertainties and steps in processes which can alter the intended effects. We found, however, a range of good and poor practice in the theories of change we examined.

8. We saw contrasting approaches to education theories of change in DFID's Nigeria and Pakistan programmes. In Pakistan, DFID's approach comprised financial aid, extensive political dialogue, technical support, funding for research and innovation and a set of interventions designed to generate social pressure for change. In practical terms, the programme involves a series of interventions collectively designed to improve learning outcomes. These include a drive on teacher attendance, the introduction of lesson plans, teacher performance incentive schemes, upgrading of school facilities and work with the examination board to raise the standards of assessment. In contrast, we found that DFID's approach in Nigeria targeted the improvement of the education system as a whole, without prioritising either the ultimate objective of pupil learning or support for weaker schools.

9. DFID's rural livelihoods programme in Odisha, India was designed at a time when theories of change were not standard practice. Despite this, DFIDDepartment for International Development India undertook a similar process, using detailed analysis of the local context, in-depth discussions with intended beneficiary communities and lessons learned from other projects, that gave the project a strong chance of success.

10. Multilaterals: In our 2011-12 annual report, we noted the difficulty of attributing impact from DFID's funding of multilateral organisations. Having undertaken three reports on DFID's relationships with and work though multilaterals this year, we observe that DFIDDepartment for International Development works with them in very different ways and that it needs to be clear what it wants to achieve in those different scenarios.

11. For example, there are aspects of the UK's engagement with the European Union and the Asian Development Bank which can only be achieved through a mixture of dialogue, influencing work and funding decisions at the headquarters level and we made recommendations for maximising the impact of these activities. DFIDDepartment for International Development also, however, co-finances multilateral projects in-country. Our reports on the Asian Development Bank and UNICEFUnited Nations Children's Fund show that the Multilateral Aid Review analysis is only a starting point for allocation decisions and that DFIDDepartment for International Development needs to understand each organisation's strengths and capabilities at a local level before committing funding.

Delivery

12. Country office capability: DFID's devolved model and the wide range of delivery mechanisms it uses to implement its programmes mean that the capability of each country office or team is vital. This is especially true in fragile and conflict-affected states, where it may be harder to achieve impact via the usual programme management practices. The evidence suggests that, in difficult environments, DFIDDepartment for International Development needs to examine carefully its HRHuman Rights policies, its access to local skills and the balance it strikes between internal and external capacity in a systematic way.

13. Following observations we have made in this area in Year 1 reports, several of this year's reports continued to pose questions about whether DFIDDepartment for International Development has the staff capability it needs in the right areas. We have found this to be a recurring theme. In particular, our report on the Horn of Africa humanitarian response noted several human resource challenges. First, the amount of time humanitarian advisers had to spend providing information to headquarters during the crisis reduced the time they had available to spend in the field. Second, additional staff were deployed on short-term arrangements, which, in our view, could have been longer, given the chronic situation and the cyclical occurrence of drought in the region.

14. DFIDDepartment for International Development has responded to its evolving business model by changing the range of skills on which it can draw through training, recruitment and new posts or cadres, such as commercial advisors or regional conflict advisors. Our report on DFID's use of contractors noted some shortcomings in programme management capability and staff continuity which led to poor mobilisation of contractor-run projects. We are, nonetheless, supportive of DFID's plans to expand its network of commercial advisers as one way of mitigating these issues.

15. Length and efficiency of delivery channels: DFIDDepartment for International Development faces real challenges and trade-offs in designing delivery channels of appropriate length. Our report on Nepal showed some of the advantages of longer delivery chains, including the ability to reach further into communities and build local capability. Similarly, DFID's work in Odisha demonstrated effective partnerships with national and state governments and with local communities to deliver sustainable results. In Sudan, we saw an over-complicated UN mechanism and a failure to shift approach, albeit in difficult political circumstances, from a crisis to a chronic situation. This way of working has been unable to deliver the desired returns.

16. Our report on Pakistan identified much good practice in designing and managing programmes to fit the local context. Based on an awareness of the challenging operating environment, DFID's humanitarian programme in Pakistan showed: rigorous and systematic fiduciary controls; clear and appropriate beneficiary selection criteria and a high degree of transparency in beneficiary selection; hotlines and complaints procedures; and robust procurement processes – all backed up by regular DFIDDepartment for International Development staff visits. These actions seemed to be key determinants of the successful impact the programme was delivering.

17. In Pakistan and Nigeria, we observed the need for partner governments to play their part if programmes are to realise their objectives. We noted some real challenges in situations where DFID's plans run alongside those of partner governments and depend on them delivering resources. Partner governments are not contributing as planned to the Pakistan health programme, nor to the Nigerian education programmes, whereas DFID's education work in Pakistan is carried out alongside meaningful government activity.

18. One recurring concern arising from both bilateral and multilateral expenditure is the lack of clarity about administrative costs – both how to define them and also understanding the full extent of them. Understanding the stated administrative costs of the first partner in a delivery channel does not, in our view, provide good enough oversight on behalf of taxpayers or intended beneficiaries, especially when visibility of those costs is possible and does not require significant additional bureaucratic effort. In our report on UNICEF, we noted a lack of transparency about how UNICEF's 7% cost recovery charge was spent and acknowledged DFID's efforts to push for greater transparency. In our report on Sudan, we also noted DFID's efforts to understand the drivers of its multilateral partners' high overheads.

Impact

19. Achieving sustainable outcomes: One way of achieving our principal objective of maximising the impact of UK aid is to examine the degree to which DFIDDepartment for International Development and other UK Government departments achieve sustainable outcomes and real impact for the poorest people. Our report on Odisha, reviewing the evidence of impact nearly two years after the project had been completed, provided evidence of positive and lasting outcomes for communities. As a result of the project's success and the partnership approach with Indian institutions, national and state governments have subsequently adopted similar approaches across other parts of India. This is an example of real, sustainable impact.

20. In chronic situations, such as Sudan and the drought-affected areas of the Horn of Africa, the challenge for DFIDDepartment for International Development is to find ways to deliver more sustainable outcomes, even where the environment is difficult. The recurring food crises in the Horn of Africa mean that DFIDDepartment for International Development has to start building closer links between its regular humanitarian programmes and its development programming. Deciding how to respond to the water and sanitation needs of the various population groups in Sudan is another example of the difficulties facing DFIDDepartment for International Development in situations which can fluctuate between open conflict, with significant population movements and calmer periods which sustain semi-permanent settlements.

21. Impacts from beneficiary involvement: Our reports on Nepal, Sudan and Odisha each highlighted the positive impact of effective beneficiary involvement in the design and implementation of the projects themselves. In Nepal, for example, we saw for ourselves the widespread pride among communities in relation to the reconstruction of police stations. This work met their needs for law and order and was directed by their active involvement, by means of local supervision committees. Similarly, the mediation programmes, which were run by local people, were making a real difference to community cohesion and under-pinning the post-conflict peace process as a result.

22. In Sudan, DFID's delivery partner (Tearfund) was able to focus its Integrated Relief and Recovery programme on sustainable outcomes by building exit strategies into its engagement with internally displaced persons and rural communities. We recognise that results like these take time to achieve but have seen how much engagement with local populations results in a real sense of ownership and improved outcomes for intended beneficiaries.

23. Impact though multilaterals: we found a mixed picture on impact delivered through multilateral organisations, with the European Union scoring an Amber-Red and the Asian Development Bank and UNICEFUnited Nations Children's Fund both scoring Green-Amber in this category. Multilateral expenditure frequently offers an opportunity for DFIDDepartment for International Development to deliver at greater scale and in sectors where it does not usually operate, such as road-building and infrastructure. A key question we have grappled with in our reports in this area has revolved around the appropriate level of DFIDDepartment for International Development oversight required to assess impact.

24. In our report on the Asian Development Bank, we drew a distinction between DFID's role as a shareholder – in which capacity DFIDDepartment for International Development has a positive influence on the Bank's strategy, policy and internal reform – and its role as a co-financier, where DFIDDepartment for International Development could provide greater support during project implementation. Evidence of impact through the European Union was mixed, with some good prospects, for example, for road-building schemes but weaknesses in performance management systems, which made it difficult to provide overall assessments of impact.

25. What does impact mean? We have been struck by the number of internal and external discussions that we have had about the concept of impact, what it means and how to measure it in an international development context. We have, therefore, decided to follow up our VFM and Effectiveness report with a Year 4 report looking specifically at how best to deliver and monitor impact. This is likely to address the key questions listed above, using an evidence base made up of our completed reports, stakeholder engagement and some new work looking at how we can provide the taxpayer with assurance that impact has indeed been delivered.

Learning

26. Sharing knowledge: Several reports have confronted the issue of how DFIDDepartment for International Development ensures that it learns and shares knowledge appropriately, its devolved structure and the use of a wide range of multilateral, private sector and NGONon-governmental organisation partners, making this a particular challenge. We are planning, therefore, to undertake a report dedicated to this topic in Year 3. Lessons already identified include:

•    the risk of contractors holding the detailed knowledge of particular projects, as we saw in our report on DFID's use of contractors, which can mean that DFIDDepartment for International Development itself does not have the insights to feed learning back into future programming decisions;
•      our Horn of Africa report showed strong evidence of learning from previous interventions within each of the affected countries, as well as from other humanitarian interventions. For example, the use of cash transfers in Somalia was an innovative approach for the region which may change how humanitarian aid is delivered in general in future;
•     the large of number separate relationships DFIDDepartment for International Development has with UNICEFUnited Nations Children's Fund makes it difficult for the former to share its collective experience of working with the latter. To address this issue, DFIDDepartment for International Development is putting in place new central posts to support country offices in their dealings with UNICEF; and
•    the general difficulty of sharing lessons and experience between country offices and central teams. Our forthcoming report will address this challenge.

27. Changing course: Our report on Nepal showed that DFID's peace and security programme had initially established relevant and realistic objectives that were consistent with both UK policy and international good practice. We found that, by the time of our fieldwork, DFIDDepartment for International Development had not adapted the direction and aims of its programmes to reflect the changing political context and that lesson-learning for the future was poor. We recommended that DFIDDepartment for International Development improve this to avoid undermining the impact achieved to date.

28. Institutional memory: The issue of maintaining institutional memory alongside the inevitable staff churn in all teams is a real challenge, particularly in fragile and conflict-affected states where staff postings tend to be shorter. We have seen a pressing need to ensure that, whatever kinds of programme or relationships are involved, DFIDDepartment for International Development needs to make sure that proper records are kept and that meaningful handover of documents, relationships and institutional memory occurs. Our Nepal report, for example, revealed a lack of up-to-date planning tools and weak project information management, which made changes in personnel difficult to manage. Commissioners also observed, on a number of their visits, that key information and insight relating to particular projects had often been lost because of recent staff changes.

________________

Source: http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-Annual-Report-2012-13.pdf