Andrew Mitchell, the UK Secretary of State for International Development, has repeatedly stressed the importance of demonstrating results and value for money in the UK aid programme. This is so obviously right that it is surprising to find some development professionals shifting uncomfortably in their seats. Why should this be? And what can be done to reconcile divergent views? Read my contribution, and join the debate.

First, it is clear that Andrew Mitchell is serious in his commitment to results – and that he is right to be so. In his first big speech as Secretary of State, delivered in Washington in June, Andrew Mitchell said ‘we’re also fundamentally redesigning our aid programmes so that they build in rigorous evaluation processes from day one. The focus will be on outputs and outcomes rather than inputs. In these difficult economic times donors have a double duty, a responsibility to achieve maximum value for money: not just results but results at the lowest possible cost.’. In that speech, he made a point of praising instruments like ‘cash on delivery’, pioneered by the Center for Global Development in Washington. He also repeated his commitment to set up a new independent aid watchdog, now formalised as the Independent Commission for Aid Impact. This will be independent of Government and report directly to Parliament through the International Development Select Committee.

It always was important to have an impact and be cost-effective, but the phrase ‘in these difficult economic times’ has special resonance. The international development budget is one of only two areas protected from cuts, at a time when other departments face //www.direct.gov.uk/prod_consum_dg/groups/dg_digitalassets/@dg/@en/documents/digitalasset/dg_191696.pdf" data-mce-href="http://www.direct.gov.uk/prod_consum_dg/groups/dg_digitalassets/@dg/@en/documents/digitalasset/dg_191696.pdf">cuts of 25% or more, the largest since the Second World War. Recent research by the Institute of Development Studies suggests that 63% of the population believe that international development should carry its share of the pain. The same IDSInstitute for Development Studies, Sussex research suggests that 52% of the population think that ‘most UK aid to developing countries is wasted’. If public support for development is to be maintained, then demonstrating that aid works will be essential.

Investment in planning for and proving results has been stepped up markedly. The Millennium Development Goals themselves provide a results-based framework. Many agencies adopted results-based approaches from the mid-1990s. Developing country governments have been encouraged to adopt results-based budget frameworks, reviewed in a series of publications by ODI’s Centre on Aid and Public Expenditure. And donors have invested more in evaluation, to close what Nancy Birdsall has described as the ‘evaluation gap’. A recent example is the creation of 3ie, the International Initiative for Impact Evaluation, based in New Delhi, but with a new office at the International Development Centre in London, launched at a conference on evidence-based development policy on 11 October.

From a methodological point of view, rigour is the watchword, with much of the running being made by survey and modelling-based approaches, such as those applied in the work of Orazio Attenaso and his colleagues at the Centre for the Evaluation of Development Policies (EdePo) at University College, London. Randomised Controlled Trials are much in favour, as exemplified in the work of the MIT Poverty Action Lab. Qualitative methodologies are also recognised as useful, however. The EPPI-Centre at the Institute of Education in London specialises in ‘systematic reviews’ which draw on a range of evidence sources to assess policy-relevant questions. DFIDDepartment for International Development has launched a whole programme of systematic reviews, the most recent jointly with AUSAID and 3ie.

There is no doubt that all this enthusiasm and energy is productive in generating useful conclusions. At the London launch of 3ie, examples included evidence on the impact of child-oriented cash transfers in Mexico, recovery from earthquake in China, water purification in Africa, and smoking during pregnancy. Evidence-based policy-making is obviously easier to achieve when there is evidence available.

Why, then, the seat-shifting? There are four reasons.

First, the obvious point, that policy is sometimes informed by evidence – and sometimes is not. This was a recurrent theme at the London conference, with presentations from Mexico and South Africa, among others, demonstrating the gap between evidence and policy; and is an issue that lies at the heart of work around the world on bridging research and policy. See, for example, ODI’s RAPID programme, including the Evidence-Based Policy Development Network. This is not to say that evidence is unnecessary, but it does mean that an understanding of the policy process is critical to effective use of evidence in policy-making.

Second, more seriously, critics of a results-based approach worry that it is too simple. The assumption, they argue, is that objectives can be defined in advance and the inputs required identified: and that when the inputs are provided, the results are expected to follow. This model is implicit in the logical framework, a hierarchical planning tool used extensively by donor agencies in project design – and one which, critics argue, underplays the complexity of real-life situations. The critics have been vocal for many years. Albert Hirschman published Development Projects Observed in 1967, writing about the principle of the ‘hiding hand’, and the importance of the unexpected outcome in development projects. By the 1980s, ‘process planning’ rather than ‘blueprint planning’ had become the accepted approach to programme management, building on lessons learned from the failure of large-scale and over-determined rural development projects. Robert Chambers was an important influence, for example through his 1983 book ‘Rural Development: Putting the last first’. More recently, the field has been influenced by work on complexity theory, for example by Ben Ramalingam and Harry Jones, and by Chambers’ own work on ‘adaptive pluralism’ as a paradigm. Outcome mapping is an evaluation and learning tool developed to capture complexity in development work, very different in style to standard quantitative approaches.

A third critique is that a results-based approach, especially of the kind implicit in the logical framework, misrepresents the purpose and focus of development interventions, which are – or need to be - as concerned with institutions and political processes as with welfare outcomes. To put this another way, the argument is that long-term improvements in welfare only result from long-term changes to institutions. Assuming that, say, improvements in child health can be delivered simply by means of vitamin supplements, misunderstands how family structures and Government health services need to change in the long term. The importance of institutions is now mainstream in development thinking, thanks, for example, to the work of Daren Acemoglu. The specific application to thinking about how aid can make a difference is found in the work of researchers like Ros Eyben, Jonathan Glennie and David Booth, who write in related ways about power, incentives and relationships. For example, a book edited by Ros Eyben on Relationships for Aid, ‘explores recent attempts from within aid agencies to go against the current flow of top-down results based management by learning how to build lasting partnerships that transfer power to those at the receiving end of aid.’ On this reading, it is perfectly legitimate to focus on results, but a much more sophisticated analysis is required of the means to the end, and this needs to be incorporated in logical framework and other analysis. There are implications for project design: not just vitamin supplements, for example, but also investments in health system capacity and accountability.

Finally, and this is a point I brought up at the London meeting, there is a real problem with the economics of the link between understanding what works in a developing country context, on the one hand, and, on the other hand, understanding the role of aid provided to support particular projects or interventions. In the simplest possible case, aid provided in kind - for example ARVs for HIV/AIDS, food aid for emergencies, or technical assistance provided in the form of expertise - sits outside the monetary economy, and can be counted as a direct transfer. In all other cases, the resource transfers are mediated by foreign exchange transfers and Government monetary and fiscal policy. Say, for example, the evidence shows (which it does) that educating girls has beneficial effects on child poverty, that the Government decides to increase the number of primary schools, and that it is considering aid to support the enterprise. Most of the additional expenditure required will be in the form of local currency, so the Government will either have to reallocate spending within its budget, or increase the size of the budget. If the latter occurs, there will be a multiplier effect on total expenditure in the economy, and this will cause demand for imports to rise. At this point, aid can play a role, by providing additional foreign exchange to help pay for additional imports. There is a link to the primary school programme, but it is indirect and may or may not be of the same magnitude as the initial budget expenditure. This is why it is important to study the macroeconomics of aid. The evaluation question, the ‘evidence’ question, is much more difficult that simply studying the project in-country.

A further problem arises with the economic case for funding projects, which is the problem of fungibility, identified by Hans Singer in 1954. Here, the difficulty is one of specifying the counterfactual. Maybe the Government was going to fund primary school expansion even without aid, but simply couldn’t find the money to build a new football stadium. By hiving off funding for the primary schools to a donor, the Government is able to release funds and build the stadium. The donor thinks it is funding schools. Actually, it is funding the stadium.

Of course, evaluators understand all this. Howard White, the Director of 3ie, is someone who has written extensively on the evaluation of programme aid and on the macroeconomics of aid more generally. When others have embarked on the same course, the findings have sometimes been startling. For example, the Independent Evaluation office of the IMFInternational Monetary Fund examined aid to Africa in 2007, covering the period 1999-2005. It found that in some countries aid made it possible to increase the level of Government spending, but that in many cases aid was used to retire domestic debt and/or increase the level of foreign exchange reserves, making no additional contribution to spending. In fact, 72% of aid overall did not result in increased spending, and 37% did not result in increased imports. These are challenging findings, but they do not invalidate providing aid to the countries in question. When domestic debt payments are high or reserves are low, using aid in this way may well be the right thing to do. What the findings do do is invalidate a narrow, results-based project focus.

Bluntly, the point I made at the London meeting was that the big investment in results-based evaluation may or may not tell us what works in domestic developing country policy – it is obviously extremely helpful in that context. It absolutely does not tell us, on its own, what works in aid policy.

Put all this together, and a strong case can be made that the current enthusiasm for results-based aid management can be conceptually and programmatically over-sold. The approach can be too simple. It can focus attention on the wrong aspects of the development process. It can misrepresent the contribution of aid to the economy. If taken to extremes, it will take aid back to the project-driven mantras of the 1970s, with all the high transactions costs, perverse incentives, and long-term ineffectiveness that have been the focus of so much remedial attention over thirty years.

Yet Andrew Mitchell is ‘obviously right’, for the reasons given earlier. The aid programme needs to be effective and cost-effective, and it needs to be seen to deliver results.

What, then, can be done? There are two kinds of answer, two ways forward.

The first answer is to ‘do aid right’ or perhaps for a Secretary of State from the Conservative Party, we should say ‘do aid centre-right’. That means focusing on results at the country level, fostering ownership and strategic planning by developing country governments and their citizens, and supporting domestic efforts wherever possible with untied budget support. This is a familiar agenda which traces its genealogy to the poverty focus of the early 1990s, the linking of debt relief to the preparation of Poverty Reduction Strategy Papers in the late 1990s, the adoption of the Millennium Development Goals, and the Paris Declaration on Aid Effectiveness, with its emphasis on ownership by countries and alignment of donors behind country priorities.

There is currently a great deal of debate about almost all aspects of this agenda, but the basic principles remain sound. Government ministers in developed countries should resist siren voices calling for the creation of yet more special purpose vehicles and global funds for particular purposes, and for the diminution of budget support. Philanthropists large and small should be subjected to seminars on the economics of aid, and should be whipped into line on Paris principles. The same goes for non-traditional official donors, and for the private sector. An exception can perhaps be made for NGOs, because citizen to citizen relationships make an important contribution to building the development case, and are generally too small to have macro-economic impacts. NGOs also usually understand better than official donors how important political relationships can be; and can do more than official donors to build voice and accountability at local level.

A new understanding has implications for evaluation and results-based assessments. Both matter. Both need greater investment. Both need to inform policy. Intervention studies are good, and so are systematic reviews, provided they assess not just the linear input-output relations, but also the political and social pathways and the unintended consequences of projects and programmes. Randomised, controlled trials have a part to play. Outcome mapping and other interrogative methods also have a place. More important, and recognising the long lines of causality linking the provision of aid to development outcomes, the best way to approach evaluation is through a country lens, setting the impact of aid alongside the influence of local drivers of change. ODI’s recent reports examining MDG case-studies, part funded by the Gates Foundation, illustrate this approach. The studies are careful to provide a balanced account of success factors, both internal and external. They set aid in context. They provide a model of how a results-based approach might be taken forward in the future.

The second part of the answer, the second way forward, is to construct a better narrative to engage the public. It does the public no service to over-simply the complexity of development and of aid impacts. Writing about this a year ago, for a meeting of DACDevelopment Assistance Committee (of the OECD) Development Communicators, I identified five ‘paradoxes’ of development communication. They were:

i. The complexity paradox.

ii. The altruism paradox.

iii. The attachment paradox.

iv. The pooling paradox.

v. The paradox of ambition.

The complexity paradox says that the simple stories needed for communication can obscure real world complexity. Make Poverty History, itself a brilliant slogan, with an objective, a doing word and a timeline, had three themes, viz debt, aid and trade. It was often difficult for journalists to go deeper. The MDGsMillennium Development Goals (see the paradox of ambition) worked as a political project partly because they provided a vehicle for human stories and personal involvement: think Comic Relief, or Sarah Brown on maternal mortality. How can more complex stories be communicated, about debt, aid and trade, about the MDGs, but also about e.g. the complex politics of Afghanistan and other fragile states?

The altruism paradox says that compassion may be most at risk just when it is most needed. This is the challenge facing development ministers in 2010, trying to protect overseas spending when local services are at risk of being cut. Polls usually show that support for development is wide but shallow . . . It will be important to continue tracking public opinion as public expenditure cuts begin to bite.

The attachment paradox says that public support for development depends on links and relationships, but these may cause mis-direction of resources. A good example is aid from the UK to India, now a middle income country and therefore on a trajectory to inegibility for aid, but at the same time a country with strong links to the UK, including but not only among the population of Indian origin. The Pakistan earthquake demonstrated similar links. The tsunami led to a hugely greater response than many other crises, with a bias in spending to countries people knew (Thailand, Sri Lanka).

The pooling paradox says that multilateral action and donor pooling (e.g. budget support) may be efficient, but that the public likes to see a flag and also direct results in return for their tax spend. Actually, public opinion surveys often suggest that Government to Government aid is often viewed with suspicion (perceived inefficiency on the donor side, perceived corruption on the recipient side), but there is a strong attachment to NGOs. Some UN agencies (UNICEF?) are exempt from the suspicion of official aid agencies. In general, the public likes to see a direct line of causality and accountability between the spending of their tax money and e.g. lower infant mortality. How can this be done when donors are encouraged by the Paris/Accra principles to pool money, and when lines of causality are inevitably long and usually obscure?

The paradox of ambition says that unachievable targets may be necessary to fire up public enthusiasm. The MDGsMillennium Development Goals have proved to be highly successful as a political and mobilising strategy, despite many problems with scope and coverage, and a level of ambition that was always likely to leave Ministers exposed as the deadline approached. They are not especially useful in operational terms, either. The former DFIDDepartment for International Development Chief Economist, Prof Adrian Wood, used to say that ‘we should take the MDGsMillennium Development Goals seriously but not literally’. Is there a better way of managing public expectations and simultaneously delivering better performance?

One year on, some of the circumstances have changed. Public opinion has probably become more sceptical, for example. Nevertheless, the paradoxes still hold.

It is possible from the analysis to construct some principles for better communication, in the form of a ten-point programme:

• Take people on a journey, from simple images and stories to more complex understanding: ‘move up the value chain’.
• Construct a narrative which is about both altruism and self interest: development as morally right, but also as necessary for our own well-being.
• Don’t just talk about risks, but also about opportunities.
• Emphasise ‘whole of Government’ approaches, not just aid.
• Tell the story that the ‘multilaterals are us’. Individual bilateral agencies don’t have to be everywhere.
• Recognise that choices have to be made and decisions taken – don’t hide from them.
• Principled decision-making needs principles – spell them out.
• Win the argument on what the problem is before trying to win the argument on solutions (e.g. climate change)
• Set realistic targets and use them pragmatically: ‘what are we going to do on Monday morning?’.
• Make sure leadership comes from the top on all the above. Ministers’ speeches are a powerful vehicle.

Much of this is already in train. For example, Andrew Mitchell’s vision of ‘One World Conservatism’ provides a set of principles and also tells a story about interdependence. His recent pronouncements on wealth creation also challenge a social welfare and narrow MDG vision of what aid can help achieve. He is certainly robust on corruption, good governance and institutions.

What is left is perhaps to think systematically about the two ways forward – doing aid right and telling the right story. Perhaps even a systematic review? This could lead to: breathing new life into the Paris agenda; offering more budget support; standing out against special purpose vehicles and vertical funds; providing more support to multilateral agencies; and remind the public that the good outcomes we all want to see are the result of multiple and mutually-reinforcing actions by many actors, a group which includes aid donors but also many others.