Reflections on ICAI’s first year
The Independent Commission on Aid Impact was announced by Andrew Mitchell in a speech to Oxfam in June 2010, and became operational in May 2011, with a mission ‘to provide greater independent scrutiny of UK aid spending, thereby maximising its value for money and impact.’ Andrew Mitchell framed this in the wider context that ‘transparency, accountability, responsibility, fairness and empowerment will be our watchwords’. The announcement was linked to a commitment to the UK Aid Transparency Guarantee. With specific reference to independent evaluation, Andrew Mitchell said:
‘Independent evaluation of British aid is absolutely crucial. There is something a bit too cosy and self-serving about internal evaluation. Reviews that focus on process and procedure miss the real issue: what did the money achieve? What change resulted from it? How were lives made better? We need a fundamental change of direction – we need to focus on outputs and outcomes, not just inputs. Sweden has been using independent evaluation for years and others, including the MIT Poverty Lab, have shown that we can be much more scientific about measuring what works. Aid spending decisions should be made on the basis of evidence, not guesswork. . . We will never maintain support for our growing aid budget unless we can offer to the British public independently verified evidence that it is being well spent.’
Two years on from the speech and one year on from the formal launch of ICAI, it is possible to see whether progress is being made. ICAI has published eleven reports, on topics ranging from programme management in Afghanistan to election support through UNDP, to education programmes in East Africa (see Box 1). It is shortly to publish its first Annual report, and to be cross-examined on its work so far by the International Development Select Committee of the House of Commons in the UK Parliament. A declaration of interest: I am a paid specialist adviser to the Committee.
A selective approach by ICAI
The first thing to say is that ICAI does not pretend to offer a comprehensive analysis of DFID’s performance, from either a results or operational perspective. It does not answer directly a question I posed back in January 2011, ‘what kind of shape is DFIDDepartment for International Development in?’. Instead, ICAI is selective in its approach.
This is in contrast, for example, to the work of the Independent Evaluation Group of the World Bank, which carries out many specialist studies, like ICAI - but which also attempts a comprehensive and systematic overview. For example, its most recent annual report, for 2011, reviews all country programmes and projects from the perspective of relevance, efficacy and efficiency. A typical comment, in this case on expanded economic opportunities for the poor, reads as follows:
‘In FY08–10, 85 percent of all WBG operations aimed to help expand economic opportunities. Among 64 country programs reviewed in FY08–11, objectives relating to expanding economic opportunities were substantially achieved in 69 percent. Eighty percent of Bank-supported projects that aimed at expanding economic opportunities completed in FY08–10 had satisfactory project outcomes.’
ICAI will surely say that this kind of synthetic overview lies outside its current mandate, but DFIDDepartment for International Development might reflect on the fact that its own published data do not include systematic project scoring of this kind. The formal DFIDDepartment for International Development Annual Report does report on results at country level and globally (with some brave assumptions about attribution, as I have argued previously, in connection with the Bilateral Aid Review), but not on project performance. I am reluctant to suggest loading more work on to DFIDDepartment for International Development officials, but ICAI itself has produced a useful report on value-for-money, which DFIDDepartment for International Development could apply to its own work.
Despite the partial nature of its coverage, ICAI has provided a range of snapshots over the past year: bilateral and multilateral; governance and economic; substantive and management-focused; Africa and Asia. Its reports are short, and it is not always clear quite how much work has gone into them; but they appear well-informed and analytical. They are certainly readable. For my taste, there are too many focused on early stage interventions, not yet ready for evaluation; and there is too much on management and fiduciary controls, not enough on impact. There is also very little on policy work: the World Bank report, for example, focuses on the spending work of the Bank, rather than its advisory role. These biases can be corrected in later years.
The scope and cost of ICAI reports
On the question of the reports being ‘short’, this is mainly about presentation (and short is good). However, it is worth making the point that these are ‘evaluation’ reports, not ‘research’ reports. In other words, there is no primary research involved, and no measurement, of the kind carried out by the MIT Poverty Lab and praised by Andrew Mitchell (for my views on that approach, propagated by Banerjee and Duflo in their recent book, Poor Economics, see here).
There was a discussion about this when the Chief Commissioner of ICAI, Graham Ward, and the Permanent Secretary of DFID, Mark Lowcock, were interviewed by the International Development Select Committee in December 2011. The Chair of the Committee, Malcolm Bruce, remarked that
‘For those of us who have to process the reports on top of everything else, they have the virtue of being short and snappy reports, if they do the job. But on the other hand, people can say they are, in the quote I have here, "quick and dirty"; in other words they are too short, too concise. It does not tell you how much went into the report. You have a policy of having a short report, but you do not know how many person days were involved.’
Graham Ward replied as follows:
‘I can certainly tell you how many person days were involved in terms of the different reports. I hope that they are not dirty; we were certainly not quick in putting them together. ICAI’s Approach to Effectiveness and Value for Money was 99.25 days; DFID’s Approach to AntiCorruption was 287.25 days; DFID’s Climate Change Programme in Bangladeshwas 144 days; and DFID’s Support to the Health Sector in Zimbabwe was 161.5 days. Those are the numbers of days that were taken by the contractor to do the fieldwork. There was then, of course, a considerable amount of input that came from ICAI’s own secretariat and from the commissioners personally.’
The Permanent Secretary, Mark Lowcock, commented later in the session that
‘On the evaluation department, that used to be the bit of DFIDDepartment for International Development from which we ran our programme of independent evaluation studies. At the time ICAI was established, we closed that business down. We do not ourselves, from the central department, produce those internal independent evaluations any more. We have recycled the money from that operation into other things. We still have a small team at the centre that deals with evaluation, but the main thing they do is provide advisory services to the several dozen evaluation specialists who are dotted around the wider Department, who commission, for example, randomised control trials and a lot of the longer running research and evaluation programmes of the sort that Mr Ward explained are not really within the resource environment or the mandate of ICAI. We have closed down the bit of the organisation that used to do what Mr Ward’s team now does, but we are still inside DFIDDepartment for International Development financing more, especially longer-term evidence generation and evaluation material, than we have ever done in the past.’
These exchanges confirm that ICAI is not expecting to generate primary evidence on impact or value-for-money. From evidence given in December, it appears that its reports may be costing up to £200k each, perhaps fewer as the number carried out per year rises, and the fixed costs of ICAI are spread more thinly. Mark Lowcock commented that
‘the cost of evaluations of the sort that are maybe broadly comparable with ICAI’s reports varies between something like £100,000 and £150,000 if we do them inside the Department. That is not way out of line with the ICAI numbers. If we are doing much more complex evaluations, for example of the sort involving randomised control trials,. . . , the cost can be significantly higher there. If you are running a randomised control trial over several years, affecting tens of thousands of people, that is obviously very expensive, but as a kind of core starting point for those complex evaluations, they might cost around £250,000.’
Personally, I would be impressed if a full-scale RCT could be conducted for £250k, at least by UK-based researchers. In any case, these exchanges establish ICAI as carrying out mid-range evaluations, more detailed than classic DFIDDepartment for International Development output-purpose reviews, considerably less than research studies,
ICAI’s judgement on DFID’s track-record
Overall, DFIDDepartment for International Development comes out reasonably from the first year set. Six reviews award an overall rating equivalent to 2 on a four point scale. Four award a rating equivalent to 3. One (on DFID’s approach to effectiveness and value-for-money) had no score. There are no scores of 1 (green) and none of 4 (red): this may reflect real performance, but may equally be the result of the evaluators’ unwillingness to be really outspoken. If this was a University examination process, the external examiners would encourage internals to make more use of the tails.
There is plenty of interest in each of the reports: how to work around the Government in Zimbabwe, for example; or how to partner with a commercially-sponsored private foundation like the Nike Foundation. More interesting, is to read the reports as a set, and identify cross-cutting issues. Leaving aside the fiduciary preoccupations which seem to loom preternaturally large in ICAI’s world view, there are five of these which caught my attention.
What is ‘impact’?
First, the reports contain interesting insights into the much-debated question of what should be considered ‘impact’ in aid evaluation: should evaluation only be concerned with final outcomes, like ‘educational accomplishments’, or should it be concerned with intermediate outputs, like ‘numbers of children at school’, or with more indirect outcomes, like the strength of the Ministry of Education and other education institutions? The right answer is ‘all three’, but sometimes the emphasis on governance and institutions is submerged in what Andrew Mitchell has called ‘bean-counting’. In support of the wider view, I have drawn a distinction between Fordist and post-Fordist approaches to results, or Results 1.0 and Results 2.0.
ICAI is sensitive to post-Fordist approaches, as exemplified by its approach paper on effectiveness and value for money. In some of its country work, it emphasises the need to examine educational outcomes as well as numbers in school, for example in its review of education programmes in Tanzania, Ethiopia and Rwanda. In its review of health and education in India, it goes further. In Bihar, the primary contribution that DFIDDepartment for International Development makes is not, according to ICAI, the financing of services, but rather support to the political process of reform, the design of new policy, and the strengthening of institutions. Technical assistance, founded in DFID’s expertise on the ground, turns out to be more useful than money. ICAI concludes that
‘DFID’s particular contributions to improving development in India are its knowledge, skills, networks and its critical yet supportive approach. DFID’s partners in India consistently pointed out that the UK’s support was valued for more than its technical capacities . . . We are not convinced . . . that DFIDDepartment for International Development can only have influence if it is seen to provide large sums of finance at the same time. We believe that DFIDDepartment for International Development should consider spending a greater proportion of its finance to India on technical assistance.’
Similar analysis underpins the evaluation of budget support. ICAI is sensitive to the opportunities for influence that come with budget support. It concludes that
‘While it is legitimate to report on the crude financing effect of budget support, the main reporting on results should focus on transformational effects (the changes brought about by UK budget support) and should capture changes in the quality of services provided (real impact on citizens).’ (para 2.76)
It will be important to pursue this nuanced line of thinking in future reports. At the ODI/IDS workshop on results which led me to reflect on post-Fordist appraoches, I observed that ‘there was enthusiastic engagement with the idea that better information was needed on results – and also lots of talk of social process, beneficiary perception, learning-by-doing, unexpected consequences, and what was described as the ‘excess certitude’ associated with technocratic approaches to results’. There’s a challenge to ICAI!
Buying a seat at the table
If influence matters as much as or even more than money, a follow-on question is whether money is needed at all. This is a question which preoccupies ICAI in a number of its reports, and which is particularly relevant in countries which could in principle mobilise their own resources. India is a case in point. ICAI reports that ‘DFID staff often argue that, if the UK wishes to influence change, it needs to provide money to ‘get a seat at the table’ with government and partners that enables DFIDDepartment for International Development to influence policies, practice and standards of financial management on a large scale’. However, ICAI concludes that ‘we are not convinced . . . that DFIDDepartment for International Development can only have influence if it is seen to provide large sums of finance at the same time’. A similar point is made in the report on budget support.
This is a question that ICAI will need to return to, and that perhaps DFIDDepartment for International Development itself will need to examine. The question could also be on the agenda of the follow-up report currently being carried out on DFID’s Multilateral Aid Review by the National Audit Office. There is a substantial literature on the policy process and on donor-recipient relationships. A recent contribution is on Knowledge, Policy and Power in International Development, by a group of ODIOverseas Development Institute (London) authors.
The staffing and skills needed to deliver high quality programmes
A related issue is that if DFIDDepartment for International Development is to engage in post-Fordist ways with institutions and political processes, then it needs the staff in place locally, backed up by professional cadres in London. ICAI make this point strongly in the India report, for example, praising the quality and level of engagement of DFIDDepartment for International Development staff, and drawing unfavourable comparisons with the establishment in East Africa. Members of the New Delhi-based DFIDDepartment for International Development health team, many of whom were locally contracted, and some of whom were on secondment from the Government, made as many as 72 visits to Bihar during 2011!
This is another topic that has long been on the agenda, and not just for DFID. David Booth is one who has long argued that donors need much greater and better trained representation on the ground if they are to engage seriously with political and institutional questions.
DFID staffing is a constant preoccupation of its friends. Numbers have fallen overall, but ingenious steps have been taken to protect ‘front-line services’, for example by reclassifying officials as programme staff rather than administrative staff. Thus, and as a result of fossicking about in DFID Departmental Reports, I was able to inform the International Development Select Committee that in 2010/11, 615 people were reclassified from admin to programme at a cost of £27 million. In the previous year, 703 people were reclassified at a cost of £32 million.
I wonder whether overall staffing and skill distribution is a question that ICAI could take up in its own right? Or perhaps it can become a running agenda item in its every report, that the ICAI Commissioners can comment on in their annual overview.
Bilateral versus multilateral aid
Asking about DFID’s staffing rather begs the questions about whether DFIDDepartment for International Development should have its own technical capacity or rely on that in the multilateral agencies. Some – see the IDC report of April 2012 on the EU – have very little capacity; and if this were the only basis on which the allocation of aid as between bilateral and multilateral channels was to be decided, the EUEuropean Union could expect a miserly settlement from DFIDDepartment for International Development . Others, however, have more to offer. There is evidence of this in the ICAI reports: on electoral support, in UNDP; on working with girls, in the Nike Foundation; and perhaps, though this is not really discussed, in the World Bank.
There are other criteria in play, however. The Multilateral Aid Review, published in early 2011, identified ten separate criteria, ranging from strategic performance to focus on poor countries and likelihood of change. Other multilateral aid assessment frameworks, like the Multilateral Organisation Performance Assessment Network, MOPAN, use similar criteria.
ICAI does not systematically ask whether bilateral or multilateral channels would best be suited to achieve the range of objectives DFIDDepartment for International Development have set in different countries. However, the choice of UNDPUnited Nations Development Programme as a partner in the area of electoral support is seen as ‘credible and to an extent inevitable’. It would be useful if allocation issues could be explored more systematically in the future. This would have been useful, for example in the Afghanistan report, in which the performance of different DFIDDepartment for International Development partners is extensively discussed – but without recommendations as to the reallocation of funding.
DFID as a venture capitalist
Finally, and cutting across many of the topics already raised, is the idea of DFIDDepartment for International Development as a risk-taker, a venture capitalist. There is praise in a number of the ICAI reports for DFID’s innovation and risk-taking. There is perhaps too little sympathy for failure. As Tim Harford argued in Adapt (and see my review here), development agencies need to foster a multiplicity of experiments, so that evolutionary pressure will identify long-term successes. Remember the Palchinksy Principles: 'to try new things, in the expectation that some will fail; to make failure survivable, because it will be common; and to make sure you know when you have failed'. It would be interesting to ask ICAI to define some operational implications of this approach.
In conclusion, and to repeat, there is much of interest in the individual ICAI reports, but the real value-added is in the opportunity they offer to address cross-cutting issues. The Commissioners have an opportunity to take these up in their Annual Report, which will also give DFIDDepartment for International Development ministers the opportunity to reply. It would be helpful if all parties, including the International Development Select Committee, could contribute to focusing the debate at this higher level of aggregation.
Comments
Once again I detect a bias for hope in your informative blog. My own take on ICAI’s start is less optimistic. For example, you are satisfied with the quality of ICAI reports. Like you I find the reports concise, sensible and readable and I agree that ICAI should steer clear of experimental and quasi-experimen tal research studies for reasons outlined in my recent article in Evaluation summarized in ngoperformance.org/.../....
On the other hand, as evaluation reports, the quality of ICAI reports is mediocre at best (amber red). They neglect policy issues and do not tackle issues of operational relevance in a systematic way, i.e. they are mostly oriented to compliance with predetermined goals. They tend to rely on hearsay rather than concrete and rigorous evidence. They read as comprehensive audit reports. But auditing is not evaluation.
Similarly you choose to highlight selectivity as a characteristic of ICAI’s work programme. This is again far too generous. I would have used the term “partial and anecdotal”. Either way ICAI is not delivering on a solemn commitment: the regular provision to British taxpayers of independently validated information as to exactly how overseas aid money is being spent; whether it is being spent on key priority issues and whether UK Aid is delivering results.
Such a commitment implies collection of comprehensive evaluation information regarding DFID’s spending and its impact on regular basis so that UK taxpayers and their representatives have access to regular and credible information about the aggregate effectiveness of UK aid.
Again you acknowledge this in your apt comparison with the World Bank evaluation system. But you do not follow through by calling for DFID’s own self-evaluation system to be reformed promptly in order to generate the evidence needed to deliver a report card on the quality and impact of UK Aid. ICAI could then be tasked to review this report card independently just as auditors do when they attest to the rigour and validity of corporate accounts.
In other words, independent evaluation by ICAI should be combined with an effective internal evaluation systems within DFID. Principled cooperation should be nurtured between the internal self evaluation system and the independent evaluation function. To get the ball rolling a process evaluation focused on the quality of self evaluation might be included in the ICAI work program as a priority item.
As you know attesting to the validity of self evaluation is an integral part of the regular work program of the World Bank’s IEG. Equally the independent IMF evaluation unit is currently carrying out such a review. ICAI should follow suit in order to encourage DFID to set up a strong and credible self evaluation system of its own. Both accountability and organizational learning require such a development. In any event, as it carries out its oversight of UK aid on behalf of citizens, the Select Committee would be well advised to ask ICAI to upgrade the evaluation skills of its contractors and to hold hearings on an updated DFID Evaluation Policy document that addresses DFID self evaluation programs and methods.
Bob
It is worth measuring ICAI’s work so far against the words of advice from its predecessor committee, IACDI, set out in Annex 1 of my final annual letter to the Secretary of State (collections.europarchive.org/. ../...).
One of IACDI’s main concerns was that ICAI should recognise the complexity of evaluating the development impact of UK aid, and here I was and remain pleasantly surprised by the early ICAI reports, which do a good job in this respect.
Like Bob, IACDI, of which he was a member, attached considerable importance to the parallel build up of DFID’s self evaluation effort. I very much agree that it would be useful for ICAI to encourage this, perhaps by doing its own evaluation of the quality of internal evaluations.
We also hoped, and I still hope, that ICAI would be able to draw on its work each year to give in its annual report an overview of its assessment of the impact of UK aid – in the way that the Head of Evaluation in DFID was beginning to do in his annual reports before ICAI was established.
Where I have been most disappointed in the reports so far, like Bob, is in the lack of real hard evidence to underpin the evaluations. You too often get the impression that the evaluators have talked to lots of people and collected some anecdotal evidence, but not much more. Obviously it is right that if ICAI had more evidence from internal evaluations to build on that would help solve the problem. But in the absence of that it seems to me that there is a good deal more they could do to collect and develop their own evidence, and/or look at issues that could help triangulate on the topic, in the way that the IMF’s IEO has done in its evaluations. For example they could make much more use of properly structured surveys of opinions. Or, to give a different kind of example, I thought that in the evaluation of DFID's engagement with the World Bank they could have usefully taken a look at how other major countries like the US and France exercise influence, to see if there are lessons to be learned for the UK.
Obviously this would add to the cost of ICAI's work, and might mean doing fewer evalauations in greater depth. That might be no bad thing. There is no need to add to the length of reports: if you look at the IEO’s reports, which are much more ambitious in scale, the reports themselves are short enough: the backing evidence is there, but in annexes or supporting documents.
Final point. I hope that IACDI’s last recommendation is not lost sight of. It would I think be good practice to commission an external review of ICAI’s work once it has had 4 or 5 years of experience, looking for example at the quality of its reports and its success or otherwise in getting lessons from evaluations learned by policy makers, to see if there are ways it’s work could be improved.
My comment was prompted by your recent mention to me of ICAI's conclusion on the systems and performance of the EC (or was it that regarding the performance of DFID in managing the tranches it signs off aptly defined by the World Bnak as lender performance?.)A n interesting follow up would be to compare the systems dfid itself has in place with which to assess the quality of its expenditure. On this, I was part of a small team that provided an independent assessment of the organisational and development effectiveness of dfid 10 years ago. It's first and last. The results were lodged (a year later) - www.dfid.gov.uk/.../ev640.pdf. I remember you commenting on it. The method was largely based on a desk review of performance assessment and evaluation reports produced since 1997, supported by interviews with some DFID
staff. It was a test of what could be said about DFID’s effectiveness based on existing performance assessment material. It followed, therefore, that it was also a test of DFID’s current performance assessment systems. No great shakes and miles away from and not in the same bracket as the World Bank and Danida, for example.
Moving on, my concern with ICAI, and I make no joke here, is its value for money. A side from the mis-leading title (IMPACT?), it cost the UK taxpayer 92.5 consulting days to produce a position paper on value for money a few months after DFID itself did the same! I felt obliged to read it coz as a morally challenged consultant we are taught to oblige. I did. I found it, and how it presumably informs what ICAI do, to be of marginal added value to guidance the National Audit Office has developed over the last ten years. the four E's, optimal mix, balance and so on. Moreover, I asked myself: what is the added value of what ICAI do over and above NAO scrutiny of DFID spend? At least the NAO tries to reduce this dissonance between what DFID beleives and its behaviour on helping them apply basic concepts such as activity based costing. I rest knowing that at least DFID's evaluation dept (another 'watchful eye' paid for by taxpayers) now appear to get the difference between randomised and pseudo-experime ntal design ably supported by HMG Treasury. Sorry...