Is evolution a good model for innovation?

Review of ‘Adapt’ by Tim Harford

It's always a pleasure to read a book which is entertaining, well-researched and certain in its argument. Even better if the book has a snappy title. Tim Harford's latest, Adapt, ticks all those boxes. It confirms my view that academics have to clear an ever-higher hurdle:  the contemporary audience has an attention span geared to the 140 characters of Twitter or the 500 words of the typical blog. Well, anyway, I fear I do. Paul Collier is another in the Harford mould - which is why I described the Bottom Billion as a master-class in bridging research and policy.

Beware, however. Good writing can reveal but also conceal. The certainty with which an argument is expressed does not guarantee its infallibility. I discovered that when I read the Bottom Billion. So, with thanks to Tim Harford for the clarity of the case and the richness of the material, let's have a more careful look.

In brief, the core argument is that human progress proceeds like evolution, with many failures and only the occasional, hard-to-predict success. Government leaders, military strategists, company chief executives, and even academics are notorioulsy bad at picking winners, technological or otherwise. The reputations of many who thought they could pick winners lie drifting in Tim Harford's wake - Winston Churchill, Lyndon Johnson, Donald Rumsfeld, and the management gurus Peters and Waterman, to name just five. There are just too many uncertainties, Harford argues, to guarantee the hitting of a moving target.

Instead, and just as in evolution, Harford advocates variation and selection, whether in military matters, health care for the developing world or the management of the global financial system. An early hero in the book is the Russian, Peter Palchinksy, murdered for crossing Stalin, but the inspiration of what Harford describes as the three Palchinksy Principles: 'to try new things, in the expectation that some will fail; to make failure survivable, because it will be common; and to make sure you know when you have failed'. The first of these principles corresponds to variation and the third to selection. The second might constitute an extension to Darwin's theory, derived from Palchinksy's experience of failed Soviet-era mega-projects, and Harford's review of systemic and potentially existential failures like climate change and the global financial crisis.

The foundation of these ideas is more firmly founded than the accumulation of anecdotes might suggest. It is notable that Schumacher is cited approvingly, hinting at a preference for small size and maximum diversity.  More notable still are the references to Hayek, including his observation that 'the curious task of economics is to demonstrate to men how little they really know about what they imagine they can design'. It turns out that, for Harford, the market is the nearest thing we have to an evolutionary testbed - a great natural experiment, in which competing ideas survive or die.

Sometimes, it is true, the market does not quite work. Despite his record as the Undercover Economist, Harford can't quite bring himself to speak of market failure or of externalities, though he knows they exist. In that case, despite the difficulties, he accepts that a modest amount of regulation may be required: carbon pricing, for example, to control climate change; or forcing banks to separate out retail and so-called 'casino' banking, in order to prevent contagion and failure in the banking system.

Regulators need to be careful, however, since Harford emphasises that regulation will characteristically create perverse incentives and unintended consequences: the term 'second best' is another piece of economic jargon hidden from the reader, but the concept looms over the account of the perverse consequences of attempts in London to insist that new buildings have the capacity to generate their own energy (the so-called Merton Rule), or in the US to regulate for greater fuel efficiency in vehicles (the CAFE standards). Simple seems to be best. Harford cites with approval the maxim of the biochemist Leslie Orgel, that 'evolution is cleverer than you are'.

I think this may be too easy. First, Schumpeter and Hayek make strange bedfellows and are often at odds. Markets, after all, reward economies of scale, in production or distribution. Walmart and Tesco are not giants of the food industry for no reason. Second, regulation is both more difficult and more subtle than Harford allows (in this text anyway): the market power of the supermarket giants may indeed be seen as socially sub-optimal (for example, from the perspective of small farmers, town centre preservation societies or campaigners against food miles), but  dilution of their dominance would require careful untying and retying of a tangled knot of local, national and even global rules and policies: land use planning; food safety;  road, sea and air transport; and myriad others. I wish that Harford had talked to more specialists in regulation, like Bridget Hutter at the LSE; and that he had written an additional chapter on this subject.

More questions arise when Harford applies his core ideas in the real world, not least in the field of international development. Perhaps not surprsingly, he is a strong advocate of evidence-based policy-making, which for him boils down to randomised controlled trials and statistical analysis. He can hardly contain his excitement when writing of the potential of 'randomistas', like Esther Duflo at MIT, to deliver rigorous feed-back: for example, to show whether text books do more for school performance in Kenya than de-worming the pupils (answer, courtesy of Kremer and Miguel: they don't).  No-one is going to argue against evidence-based policy making. We are all heirs to Archie Cochrane and enthusiasts for systematic reviews (or what we used to call survey papers). However, a more measured understanding of how to evaluate social processes would not have gone amiss. There is an active debate, not acknowledged here, about balancing quantitative and qualitative approaches to analysing impact in development, and about how to move beyond Fordist approaches. This is an approach I have dubbed 'Results 2.0'. In another context, Harford would probably be sympathetic. Just as I was writing this, he published a piece in the Financial Times, mocking the use of econometric growth models to explain long-term development performance.

Harford also flirts with inconsistency when it comes to technological innovation. He is sceptical of government-led and project-focused research funding, on the grounds that it smacks of central control and trying to pick winners. Instead, he advocates open-ended funding targeted at the best researchers; and enthuses over market-based instruments like the Advanced Market Commitment, which encourages drug companies to invest in new vaccines by guaranteeing a market for those determined and lucky enough to secure a breakthrough. Even better, Harford positively drools over competitions to reward successful innovation, whether in manned space flight or to bring to market a vaccine to prevent HIV/AIDS. Never mind that a prize to incentivise work on a vaccine might have to be as much as US 10-20 bn, prizes encourage evolutionary innovation and competition in a market framework: they exploit the advantages of 'an amazing decentralised cloud computer - the markets that make up the world economy - to provide feed-back to billions of individual experiments'.

There is again a literature on this, though not one that Harford cites. William Easterly, for example, has written about the advantages of searchers over planners in allocating aid resources; and there is a respectable literature as well as exciting experience on open source innovation, which draws on the widest possible community to tackle current research problems. Taylor and la Barre’s book, Mavericks at Work, contains illustrations

But does Harford really think that research competitions are appropriate to all kinds of research problem? Does he think that $US 20 bn winner-take-all prizes will foster a research ecology of small-scale evolutionary start-ups? And does he have any understanding of how hard it is to build and sustain research capacity, especially in developing countries? We are not given the answer to these questions, but we can guess that Harford has more common-sense than his text allows. For example, he cites the work of the 'army of policy wonks' who are alleged to have shown that there is little to choose between a carbon tax and a cap and trade regime: presumably most are footsoldiers in University departments and specialised research institutes, struggling to balance research quality against the demands of grant-making bodies and Government research assessment. It is hard to imagine how a competition might be structured to mobilise these platoons; or how their institutions might find the up-front costs and be able to indemnify the losers' losses.

Take another example, Harford's favoured idea of a price for carbon. Easy to suggest, but hard to deliver, as he begins to recognise - a task requiring serious work on the theory and practice of collective action, on international negotiation and on the working of tax administrations in countries both rich and poor. A prize privileging private sector actors in a global market place for clever ideas in political economy and public administration? It is hard to imagine such a system out-competing the boring old routine of administering research grants and building collaborative epistemic communities in which ideas are debated and shared. Elinor Ostrom, remember, won a Nobel prize not a patent for her pioneering work on collective action.

Competitions have a role in research, and so, of course, does the private sector. But as with the debate about regulation, Harford would have benefited by reviewing work on innovation systems theory, for example by Chris Freeman, Andrew Barnett and others associated with the Science Policy Research Unit at the University of Sussex. He could greatly have strengthened his understanding of how to engage with users and how to manage the translation of scientific advance into concrete benefits. The extensive research base on bridging research and policy in social science could have been equally useful.

Do these qualifications mean that those interested in how to solve real-world problems should turn their backs on this excellent book. Absolutely not. However, they should perhaps think of it as a treasure trove of ideas scattered about the evolutionary landscape, many of which will flourish, some of which will die, and some of which, given a favourable wind, will eventually  mutate and prove to be useful. That is a suitably evolutionary approach of which Peter Palchinksy and even Tim Harford himself might approve.