This is the third in a four-part series of posts asking: What has changed since 1997 that bears on the way we should govern?
In the first post I looked at the rise of a digital practice of production, which I argued redefines what we need from the state but also changes how the state should function.
In the second and third posts I’m looking at changes to how we understand the world. What fresh intellectual material do we have to work with that wasn’t readily available in 1997?
Rather than cover the waterfront, I’m giving two examples of insights or approaches that have become more readily available since 1997.
First I looked at the revolution in behavioural science, which I think has much further to run than people realise. In this post I’ll look at advances in the way we understand and work with complex systems.
A complexity awakening
In the last two decades there have been important contributions to public policy from disciplines related to complexity, from systems thinking to social innovation theory to chaos theory, cognitive science, and artificial intelligence. This has helped us think more clearly about complex systems in the context of public policy.
These insights are themselves rather complicated but a reductive summary would be that:
Societies and economies are irreducibly complex systems;
It’s therefore important that we govern in a way that respects and works with system-level properties like boundaries, tipping points, and interdependencies;
Many social and economic outcomes are emergent. Which is to say they’re best thought of not as the sum of individual people’s actions/intentions but as properties of the system itself. So we should recognise the primacy of patterns.
These insights are not literally new since 1997. The idea of complex social systems with emergent properties has a long heritage, running back through traditions like 19th century sociology and thinkers like Emile Durkheim.
Plus the idea of complex systems has already had a big influence on public policy.
In a field like macroeconomics, for example, the 20th century was pretty much defined by debates about complex systems. Hence ideas like animal spirits and unsatisfactory equilibria, which describe how crowds behave unpredictably and how economies can get stuck in outcomes that literally no-one wanted. These ideas explain the way we do macro policy — it’s designed for complexity.[1]
But although some domains of public policy have internalised the idea of complex systems, this is the exception, not the rule. The default when we make public policy is to assume not that we’re working in and through complex systems but to act as if the world is — if only we try hard enough, and are clever enough —predictable and subject to control.
Over the last 25 years, though, we’ve experienced an awakening, as our eyes have been opened to the flaws of this approach.
Or it might be better to say we’ve been subject to a relentless education. And this education has come in three stages.
First, we’ve been hit repeatedly by policy problems that are object lessons in complexity. This has made it hard to ignore how inept we are at working with complex systems — and how costly this failure is.
Second, we’ve been presented with an increasingly awkward contrast — between the management practices of contemporary, internet-era organisations, which are designed for complexity, and the way we make policy, which isn’t. This has exposed in embarrassing detail the flaws of our ageing approach to policy-making — and to governance more generally.
Third, we’ve seen what an alternative looks like. Because in the vanguard of the public sector and civil society — in our most open, forward-thinking teams and organisations — people have started to practise a new approach to policy-making that works for complexity.
That’s the short version, anyway. Let’s spend a bit more time with each of those lessons and see if we can be more precise.
Lesson 1: A brutal education
Climate change has been one of history’s most powerful teachable moments. And ranking up there too would the related crises of ecosystem collapse, habitat destruction, and biodiversity loss.
So what have we learned?
One takeaway is that we’ve failed to behave — and to govern ourselves to behave — in a way that respects the properties of complex systems.
Throughout the 20th century we acted in ways that made sense when evaluated on their own narrow terms but that were disastrous in the context of a complex system.
Our actions were often rational when calculated as the short-term hedonic gain to an individual, or the profit of a firm, or the Net Present Value of a policy, or the GDP of a country. But when situated in the systems in which they inescapably sit, these same behaviours proved catastrophic.
The climate crisis has also been an historic moment of hubris. It’s the bill for our lack of epistemological humility.
As our environment collapses, we see how badly we underestimated the hidden order and resilience in complex systems. From deep water currents that slow or reverse when seawater warms, to mycorrhizal networks that break down in monoculture forestry and farming.
We spent decades thinking we were doing surgery — cleverly isolating variables, extracting value with precision — when it turns out we were doing butchery, severing networks that are vital to life.
So environmental breakdown has been a brutal, hold-our-faces-to-the-empty-cookie-jar education: DON’T IGNORE COMPLEXITY.
But the awakening goes broader than this.
It’s increasingly clear that ecological breakdown is only one example of a more general shift in the character of social problems — a shift towards complexity.
We’ve seen a rebalancing away from ‘technocratic problems’ — problems that can be isolated and specified, and that therefore lend themselves to technocratic solutions — to problems that are characterised by their irreducible human complexity. (More on this here.)
The clearest example of this shift has happened within healthcare.
If we go back 150 years, the main challenges in healthcare were acute disease and issues like bacterial infections — problems that could be isolated and treated with medicine.
Today, the main burden of disease sits with chronic and mental health conditions and so-called lifestyle conditions. These conditions cannot be seperated from phenomena like loneliness, or local/kinship/caring relationships, or food systems. They are social as much as medical.
So just as we must learn to work with complex environmental systems, from deep water currents to nutrient cycles, we also need to learn how to work with the hidden order in complex social systems.
And if that sounds like an indulgent point, remember that around 75% of NHS spending is accounted for by chronic conditions.
So our most urgent policy challenges — from climate change to saving the NHS — require us to learn how to work with complexity. [3]
Lesson 2: A detailed dressing down
So lesson one has been a blunt education. We’ve been bad, very bad, to ignore the complexity of the systems we live in.
Lesson two is more diagnostic in nature — it shows us why we’ve failed.
The teaching technique here has been to show us an increasingly striking contrast —between the way contemporary organisations run and the way we make public policy.
It’s now standard in contemporary organisations to work iteratively; to optimise for outcomes, not outputs; to break up work with intentional periods of reflection; and to delegate work to mixed-discipline teams that have the authority and capability they need to act with self-sufficiency.
These practices integrate learning and doing. And it’s not a coincidence that they work for complexity. The whole paradigm they’re part of (call it what you want: Agile, product working, internet-era methods) emerged in response to the rise of software as a way to deliver value to users at pace in conditions of extreme complexity.
With each year that passes, these contemporary management practices contrast more awkwardly with the way we make policy.
Take as an example the linear way we make policy.
When we make policy, we follow a series of steps. We investigate a policy problem and then we diagnose the cause, develop a list of options, and choose our favoured solution. Then a politician announces the policy and we consult and then we legislate. And finally we hand the policy over into the system for delivery. (And that is the good version.)
This policy-making process can take years end-to-end, while meanwhile the world changes beyond recognition.
But the linearity of policy-making isn’t just slow, it’s also horribly risky. Because it reflects Whitehall’s deepest design principle: the idea that policy is separate from and prior to delivery.
This means linearity also hinders learning. The most important decisions about policy design come at the start, when we know least about the problem. And the doing — the moment we really start learning — comes at the end, when decisions have already been made.
So linearity affords few opportunities for iteration. [2]
And so aside from anything else our approach to policy-making is just insanely, pathologically, risky. It means the first time we try a policy out for real it’s often already literally the law and already applies to millions of people out in the real economy.
I’ve described this before as being like learning to play the guitar by booking yourself a gig at Glastonbury, and then preparing by reading lots of books about guitars, and not touching an instrument until you’re on stage.
But linearity isn’t the only reason policy-making struggles with complexity. The policy-making process also struggles with complexity because:
It is hierarchical and often centralised, so that as decisions get more important we escalate them further away from the work.
It aspires to isolate variables rather than identifying and working with interdependencies. Hence it’s often blind to things like relationships.
It aspires to a logic of abstraction and extraction, so that our goal is normally to find universal rules or transferable insights to be ‘rolled out’. This is different to seeing policy-making as a science of how to develop and deepen ideas in context, such as in a locality.
The simplest way to put it is that policy-making is industrial. Hence all our language is about levers and machines.
And so this increasingly awkward contrast between industrial-era policy-making and internet-era methods helps to explain why we’re making such a hash of governing a world defined by fluidity and complexity.
Lesson 3: How things could be different
That’s all a bit depressing, sorry. But here’s where things get optimistic. Because we can now see what an alternative approach would look like.
For around 15 years, a new generation of reformers has been working in pockets in the public sector and civil society, practising and refining a way to do policy that works for complexity.
This work takes those insights from systems theory, complexity theory, and chaos theory. And it combines these theories with practical knowhow from digital leaders, service designers, and product owners using internet-era methods, especially in the public sector and civil society.
It adds up to what I could call, a bit grandly, a new science — or some would say a very old science — of how to govern for complexity. [4]
20th century policy-making was linear, siloed, hierarchical, and abstracting, and beguiled by illusions of precision and control. And the tools it used embodied those principles.
21st century policy-making is iterative, cross-discipline, user-centred, situated, and humble — and therefore far less risky. It works with a new and now well-proven toolkit — a new set of organisational structures, team rituals, processes, management practices, and mental models.
Those tools enable us to:
Integrate learning and doing, so that work doesn’t move in a linear relay but in cycles of learning
Start simple, de-risking the work by putting ideas out there quickly, failing as small — and learning as fast — as we can, starting by testing our riskiest assumptions
Delegate decision-making to mixed-discipline teams that sit close to the work — teams that have the authority and capability they need to be self-sufficient, and therefore to move quickly
Sharpen accountability, so that people aren’t responsible for some siloed slice of bureaucracy but for end-to-end services — for delivering outcomes to users
Emphasise people over policy — start with capability and culture, not with the ‘right’ policy answer
Think less about levers and more about catalysts or signals. For example, we use tools like challenge prizes, and regulatory sandboxes, to make space for innovation in service of a goal. And we see prices less through the lens of individual incentives and more as a way to trigger system tipping points, e.g. for the adoption of a new technology. So we take a different approach to things like environmental subsidies.
Work in the open — flushing out bad ideas, and helping the system to adapt. Which also allows us to be more porous, and less dirigiste. So we crowd in ideas from the private and third sectors.
Favour processes that are good at developing and refining an idea in context, rather than processes that are all about abstracting insights away from context. For example, by giving genuine agency to localities and applying fewer conditions to funding.
This might sound like a list of aspirational principles. But 15 years into this revolution, it’s really just a description of work that is already happening.
In an upcoming report — The Radical How — from Nesta and public.digital we’ll unpack examples in more detail. The point to note here is that the examples are not niche digital teams, or tech companies.
These ways of working have now spread to major government initiatives, such as the Universal Credit rollout, where the DWP escaped from a car crash by pivoting to this kind of approach. Or Defra’s future farming programme, where great leadership from Janet Hughes means a similar spirit is being brought to a complex policy challenge. [5] We also share lessons from Nesta, where I work, where we’re applying these methods to three missions of climate change, obesity, and early years outcomes. (More here.) So far from being a novelty, these approaches are now mature — but only in pockets. [6]
Which takes us back to where I started this part of the series, when I wrote that:
…there is now a big backlog of insights or approaches that are well-established in academia, or in expert-practitioner communities, but that haven’t yet translated into the premises by which we govern.
I think this whole question of how we govern for complexity is a great example of an approach that’s caught in the backlog.
Among contemporary practitioners, this stuff isn’t new any more — it’s just the way things are done. But what’s been missing is leadership that understands the potential of these approaches, and that has the political will, intent, and seriousness to spread them across the system.
This was the third in a series of posts in which I’m trying to answer the question: What has changed since 1997 that bears on the way we should govern?
In the first post I covered the rise of a digital practice of production and then in the second post and in this one I’ve covered two post-1997 intellectual or methodological developments: a revolution in behavioural science and an awakening in how to govern for complexity.
In the last post in the series, I’ll approach the question from another angle, which is to take advantage of hindsight.
I’ll look back on the governing philosophy or mentality that Labour brought to its last period in office and ask: what worked and what didn’t?
Here are the first and second posts in the series. To get the next post directly, you can follow me on Medium or support my writing on Substack. In the meantime, if you’re interested in these themes, try my recent posts: How to govern human or How to solve complex problems. Or my book, End State. Also a shout out to Tara Dawson McGuiness’s book, Power to the Public, which does a fantastic job of describing a similar delivery philosophy in the US context.
Footnotes
I find this example of ‘monetary policy as agile policy’ quite thought-provoking. People talk about agile policy-making as if it’s a new thing but since monetary policy internalised the idea of complexity a long time ago, it applies a lot of the same principles with none of the jargon. We enshrine an outcome in law, an inflation target; we delegate authority to the Bank of England and the MPC; we work in increments, raising or lowering interest rates; we divide this up with time reflection in six-weekly MPC meetings; and we work in the open, publishing minutes so that the system can learn and adapt. So I wonder if this gives us a helpful thought experiment: are there other areas of public policy where we could apply the same kind of approach? (e.g. GP waiting times? Hospital discharge processes? Heat pump installations/home decarbonisation?)
I know someone will shout at me here and say: ‘but what about pilots?’ Or they’ll say I’m behind the times and policy-making is now all about continual improvement. I know in some cases that happens. But surely it’s telling that in policy-making we still have discrete mechanisms like pilots for the times when we want to learn something. Plus a pilot implies a time-boxed period of learning that then ends, as you move into delivery. And so even if we sometimes make space for iteration, this feels to me like making little whirlpools in the linear river — local and fleeting. The overall process and language of policy- and law-making — and not least of legislation itself — is still overwhelmingly linear.
Home decarbonisation is another example of a policy challenge that demands new approaches. In the UK we need to install anywhere from 600,000 to 900,000 heat pumps a year and we’re currently installing c.60,000. What’s holding us up is a tangled ball of constraints: labyrinthine planning laws, sticky human behaviours, technological bottlenecks, a lack of installation capacity. It’s hard to see how we cut through these problems with linear, siloed, top down policy-making. Not least because every time we make progress we hit a new problem, often noticed first by people on the ground, like heat pump engineers. So this is another example where we need to set ourselves up in a more mission-oriented way: drive for an outcome, delegate power to a person/body accountable for the outcome, equip them with a cross-discipline team that includes people with frontline experience, work in the open, and tweak policy in pursuit of the goal.
This is a point for another day, but one thing I find most fascinating about all of this is that we’re not so much learning a new science as re-learning an old one. Namely the science of practical wisdom, or knowhow, that was described by anthropologists like Levi-Strauss, and in the anti-modernist critiques of writers like Jane Jacobs.
The UK’s experience with Universal Credit is in this sense similar to the US experience with Healthcare.gov, where a policy failure brought about a change of approach. For this and other US examples see Tara Dawson McGuiness’s great book, Power to the Public.
This is also why I have less and less patience with old charities or public sector bodies who are still caught up in arguments about agile and waterfall, or that spend their time splitting the difference. 10 years ago these organisations were lagging the digital frontier. Now they’re lagging good teams in major government departments like DWP, Defra, or DfE. So it’s less and less excusable when people running large-scale services for the public say they’re sceptical about, or don’t really know much about, what are really just contemporary leadership practices.