Can Test & Learn methods save the NHS?
On the challenge of using these methods here
Bureaucratic systems are good at cranking the handle, delivering outputs at scale, often under intense pressure, but they tend to be bad at learning and progressive improvement.
Nowhere is this more true than in the NHS, a system that every day delivers a stunning number of appointments, prescriptions, and operations, but that has been stuck for years on an array of complex problems (e.g. managing and preventing chronic conditions like diabetes, or preventing falls among elderly people), that are now threatening to sink the system.
There’s a case to be made that solving this issue — finding ways to help the NHS learn, and spread, progressively better ways to support well-being — is the most important delivery challenge facing the UK government. I’ve written about this before in the context of the government’s 10 Year Plan for Health, which rests almost entirely on our ability to snap the health system out of various sticky states it has been stuck in. And I’ve written about the conditions required for iterative working.
In this post, I’ll ask a more direct question: can we use Test & Learn methods in healthcare, and, if so, how? I want to write about this partly to challenge NHS exceptionalism — the idea that the NHS is different, and Test & Learn methods can’t work here. But I also want to name some things about the NHS/healthcare that genuinely are different, and suggest some ways that these methods can be adapted in response.
I’m conscious that this is all a bit insider baseball, so I’ll start with some definitions. Then I’ll list some ways in which the NHS/healthcare differ from other areas of public policy, and I’ll propose some ways to adapt Test & Learn methods. Finally, I’ll end with some suggestions for work that could usefully be done.
This post integrates a lot of other people’s work, and is informed by lots of insightful conversations; think of it as a curation of ideas from a whole host of people, including Rachel Hope; Richard Pope and Ben Welby; and cardiologist Dr William Bradlow, among others. So any credit is shared (and any mistakes are mine!)
1. What is Test, Learn (and Grow)?
Test & Learn is a way of working in which a mixed discipline team gets up close to a complex problem and works in rapid feedback loops of real world experimentation. It borrows from agile software development and user-centred design, but applies these methods more broadly to all aspects of end-to-end service design and delivery, and policy development, and even to governance and the wider operating environment.
The most striking example is the way the government rescued the early rollout of Universal Credit by moving away from old-school programme management to develop the new system iteratively, starting in one place with a small number of families and iterating as they scaled out. More recently, there is the Future Farming and Countryside programme, which has grappled with the daunting task of reimagining farming subsidies after Brexit. This work has been so outward-looking and adaptive that, perhaps uniquely among big government programmes, it has won praise from many of its main civil society stakeholders for being so responsive.
Compared to traditional bureaucratic management practices, the value of Test & Learn is that you can scale a service while learning, as your confidence and delivery capacity grows. This contrasts with trying to plan upfront, when you know the least about the problem. By learning as you go, you can reduce risk significantly, release value earlier, and you can also show your progress in a literal sense — you can point to something you have built, in the real world, to show what the future looks like.
So that’s what Test & Learn is. But it’s also important to say what Test & Learn isn’t. Test & Learn is not:
A form of ‘pilot’ — it is not a way to discover the ‘right’ answer so that you can ‘roll it out’.
A way to roll out a specific technology — in a Test & Learn team, the focus is on a problem, not a pre-chosen solution. The goal is a better outcome, not an artefact (e.g. a piece of technology).
A management consultancy technique. Test & Learn is a broadly applicable way of operating, and a set of management practices. It is more analogous to ‘agile’, or to New Public Management, than to any particular consultancy framework (e.g. the McKinsey 7S model)
Finally, I think it’s fair to say that Test & Learn is not a ‘lever’ in the traditional sense, because it’s not something that a person ‘pulls’ or operates from the centre of the system. However, I do think test and learn is a technique in the delivery toolkit. Or maybe it’s better to say it’s a delivery discipline, or philosophy, that offers a suite of useful techniques. And it’s increasingly vital that this suite of tools be integrated into government delivery functions. (Watch this space for some upcoming work on how we can do this.)
The Cabinet Office TLG programme
Right now, the most explicit effort to spread Test & Learn in the UK is being driven by the Cabinet Office. This is happening through a programme called ‘Test, Learn and Grow’ (TLG), and the addition of the word ‘grow’ is important.
The TLG programme is partly about supporting Test & Learn methods across the system — helping teams work in this way across the UK, in local areas, to make headway on complex problems. This is already happening for a host of tricky challenges, including efforts to: support children with special needs, get more people into work, roll out breakfast clubs, increase the uptake of Best Start Family Hubs, establish neighbourhood health services, and tackle violence against women and girls.
The addition of the word ‘grow’, however, means the work goes further than just supporting these teams. It turns Test & Learn from a delivery method into a model for public service reform and improvement. The core insight is that we can use Test & Learn teams as a discovery mechanism. The teams work close to problems, trying to improve social outcomes, and as they do so they hit up against barriers. For example, they might be blocked or slowed down by an outmoded sign-off process, or have their time wasted by a redundant risk management process, or they might realise they are missing certain capabilities or technologies. The TLG programme can then act on this information, helping to remove these barriers for everyone.
This idea of using iterative methods to drive system change is quite new and is potentially very powerful, but it is not easy to do. A lot of the work of the TLG programme is therefore about establishing the necessary architecture for listening and responding. How do you get the lessons you’re learning about barriers to the people with the power to remove them?
Stepping back, the way I think about TLG these days is that it’s not so much a ‘lever’, but is more like a good car tyre. It is a way of working that changes the system’s interface with its environment in order to increase traction. This is especially important in the slippery terrain of complex problems.
2. When are these methods useful?
Test & Learn works in a broad range of domains, but it is not universally applicable. It is less useful for work that has a well-defined technical answer and where the local context is not all that relevant. For example, it would not be sensible to undertake a big Test & Learn initiative to sanitise drinking water, since this a problem to which we already have a pretty good technical answer, and where the answer is the same everywhere. This is to say that, when it comes to sanitising water, traditional bureaucratic methods have traction.
Test & Learn methods are more useful when work is complex and social and variable, or we might say slippery. More specifically, these methods are helpful when:
There is thin evidence
There is high context dependency — where doing something in one place, or with one person, might yield different results to another place or person
We are likely to need new delivery models, technologies, and forms of collaboration, i.e. when we are not just solving a problem, but also improving our tools as we go
Our goal is behaviour change (e.g. for citizens, or for staff in a public service); especially when it is hard to test the behaviour reliably in a simulated setting
We are clear about our desired outcome (e.g. ‘boost screening take-up’, or ‘reduce administrative burden for cancer patients’), but there is no single point of intervention, so that we have to keep trying a range of small things, and recalibrating our efforts
The cost of being wrong is high, but the cost of small tests is manageable. (It would not be worth using Test & Learn to change the font on a letter, but likewise it would not be safe to use Test & Learn for nuclear security.)
There are complex trade-offs, especially between different professional standpoints (e.g. simplicity of design vs. technical complexity; or cost vs. accessibility)
I would add that the ‘grow’ aspect of TLG becomes especially important in highly regulated environments, when we can’t iterate a new service delivery model without also changing a constraining set of rules. In these cases, the mechanisms for escalating system-level insights to get rules changed are not an optional extra, they are absolutely essential.
The big takeaway is that, when problems have these qualities, a bureaucratic mode of procedure, and its associated delivery regime and management practices, is simply the wrong way to do things. It is no less wrong than using a hammer on a screw, and it will lead to similar outcomes — progress will be infuriatingly slow despite making lots of effort, and people might even get hurt. (And when we look across the public sector today, doesn’t this description feel familiar?)
3. What is different about healthcare?
The other takeaway from the list of criteria above is that healthcare contains a lot of problems like these, so there should be huge potential for using Test & Learn methods. It’s also clear that healthcare has taken on more of these qualities over time, as the burden of illness has shifted away from tightly defineable acute communicable diseases (e.g. bacterial infections) to messy and complex conditions (e.g. mental health; or the chronic and inter-related physical manifestations of ageing; or the accumulated bodily harm of prolonged unhealthy behaviours). So we can also say that using Test & Learn methods in healthcare is becoming ever more important.
At the same time, healthcare in general, and the NHS in particular, are not easy environments in which to use Test & Learn methods. So let’s turn to these differences but let’s also approach them with a problem-solving spirit, and ask how Test & Learn methods could be adapted.
In the list below I’ve called out some of the most relevant characteristics of healthcare that make Test & Learn methods difficult. And in each case I’ve suggested a way Test & Learn methods could be modified. NB: This covers a lot of ground, so think of it as nothing more than some opening reflections.
Clinical quality is paramount, and the risks of a medically inappropriate treatment can be more immediate and catastrophic than in other policy areas (e.g. compare giving someone the wrong medicine to using a bad method to teach a child to read).
Modification: The importance of clinical judgment means we need to integrate clinical expertise and assurance into the multidisciplinary Test & Learn team. Clinical assurance cannot be treated as a gate to be cleared at the end, or a process in which decisions are escalated while the team waits for answers; this kills the iterative nature of the work. Instead, Test & Learn teams must be set-up to develop clinical tests, guardrails, and approval mechanisms as part of their work.
2. The organisational landscape in and around the NHS is extremely fragmented. A single patient pathway can run through primary care, acute services, mental health, Local Authorities, and third sector and private providers. Test & Learn methods will struggle if they cannot work across organisational, contractual, and data-sharing boundaries.
Modification: A complicated landscape makes it even more important to work in the open. This is one of the most efficient ways to flush out dependencies with work other people are doing. The complicated landscape of the NHS also means it’s valuable to create ‘greenfield’ Test & Learn teams that sit outside the system’s various provider-centric siloes (e.g. don’t establish Test & Learn teams within the siloes of ‘primary care’, or CAMHS, but in the cracks between these institutions, or in the space above them). (The ultimate goal should be to move beyond the antiquated operating model of functional silos to one built around patient outcomes, but let’s take one step at a time.) Big institutions in the healthcare system should work together to establish Test & Learn teams on shared priorities, staffed by secondees from the relevant organisations.
3. The NHS suffers from by far the worst case of provider lock-in I’ve seen in 20 years working in and around government, most of all in the software market. Much of the NHS’s technology estate is built and run by legacy technology providers, often working in uncompetitive markets (e.g. see Electronic Patient Record systems, and ‘GP IT’ systems). This leaves staff and patients using inflexible technologies from the 1990s, and locks teams into the outmoded delivery methods used by old-school providers. This makes it harder to iterate.
Modification: To ease the bonds of provider lock-in and legacy software, we need to make sure Test & Learn methods are coupled with a much more determined agenda to support the exit from legacy technologies and contracts, and a more active drive to make NHS provider markets more dynamic. (As an example, we can enforce interoperability standards, give start-ups easier access with a proper digital marketplace, and build public options to break the hold of monopoly/duopoly providers.) (I describe this vital work of market-shaping more in this earlier post.) There might also be value in publishing a playbook to make Test & Learn methods explicit as a way of working in the NHS, and to make clear that providers will often be expected to work in these ways e.g. when seconded into teams in public institutions.
4. There is low digital literacy at senior levels in healthcare, partly because senior leaders often come up through clinical specialisms. This of course brings valuable experience, but it also means there are fewer native digital-leaders in healthcare than in other parts of government (e.g. in a department like HMRC). This has the very costly implication that the conditions we need for good digital work are least well advanced in healthcare, which is the part of government where they would have the most impact. For Test & Learn, this means we’re often starting one step behind, working in systems that haven’t yet even established the agile and user-centred methods from which Test & Learn originated.
Modification: This is one reason I’ve argued that the recently advertised NHS CDIO position is the single most important digital job in government. It is vital that it goes to a truly world-class talent with deep experience of digital delivery and an understanding of digital work in a public sector context. Beyond this, we need to make sure people in senior roles across the entire system have a deep understanding of the conditions required for good digital work, and more generally for Test & Learn methods (for a good start on this, see the curriculum developed by Public Service in a Digital Age). This would be a great early priority for Britain’s new National School of Government.
5. In healthcare, there are especially strong professional and regulatory barriers to change, even more than elsewhere in Whitehall. This includes deepfelt professional identities; siloed and hyper-specialised training pathways; and a culture of risk-management, indemnity, and regulation. Think of bodies like the General Medical Council, Nursing and Midwifery Council, Care Quality Commission, Medicines and Healthcare Products Regulatory Agency, Health Research Authority, or the many professionals working on information governance. These professions are, of course, important and valuable. But they have often internalised a conception of risk, and risk management practices, that is anathema to iteration.
Modification: Test & Learn methods need to navigate these long-standing cultures and respect the good reasons they developed, while also being brave enough to name when old management practices are harming patients. The highly regulated climate of the NHS also means we need to put more emphasis on the ‘grow’ aspects of TLG — using Test & Learn explicitly as a way to identify, with the relevant professions, which rules are working, and which ones are blocking innovation. As part of this, we should also learn more from teams who are already innovating across the NHS, convening innovators and giving them a direct line to senior people to help remove barriers.
6. Healthcare is especially data rich and there is intense scrutiny of information governance. There is also a culture of scientific practice that has a particular, linear view of experiments (run a trial, publish results). So although healthcare pathways generate lots of data, this data tends to be seen through a lens of research, as opposed to iterative service improvement.
Modification: Test & Learn methods must respect these cultures and work with them. Specifically, we can find ways for Test & Learn teams to collaborate with researchers, including ways for them to participate in data-sharing and routes to academic publications, and secondary uses of data. Test & Learn teams could even support these processes by helping services to generate research- and regulatory-grade data as part of their work. This would support healthcare researchers and could also help to attract investment towards promising innovations.
7. There are many professions in healthcare who already see themselves as owning the work of iterative improvement (e.g. audit, service improvement, research, etc). Test & Learn teams might run up against these professionals and jar with them, being seen as a competitor.
Modification: To avoid clashing with other professions working on continuous improvement, Test & Learn initiatives in the NHS should engage these professions as allies, making sure their work is a natural evolution, and not a competing tribe. This means making explicit connections to these existing methods, and learning from them, drawing on their expertise to help codify a healthcare-friendly version of Test & Learn methods.
Each item in that list deserves an essay in its own right, so I am really just scratching the surface. The two main points to land are (1) the default climate of the NHS/healthcare is hostile to iterative working (which is partly why the system is being drowned by complex problems in the first place). But also (2) there are practical ways we can change this, and these are all within the gift of senior people in the relevant institutions.
This is not to say that these changes are easy, and I don’t think we should pretend we can make iterative working doable for everyone across the NHS tomorrow. What is doable, however, is to gradually multiply the number of pockets in which this work can happen. The good news is that this process has already started, and Test & Learn methods have already been used safely and effectively in the NHS, typically where contemporary leadership has created the right conditions. For example:
The Manage Vaccinations In Schools (MAVIS) service successfully iterated towards an outcome (‘increase the take-up of childhood vaccinations’), rather than starting with technology requirements
The NHS partner SH24 applied a whole-service design approach to the delivery of sexual health services
The University Hospitals Birmingham Cardiology team applied a similar approach to the design of a genetic screening programme
So the goal, in a sense, is simply to spread these conditions so that more teams across the healthcare system can work in these ways.
4. Where could we start?
It’s easy for work like this to feel overwhelming. The system is so complicated it can be hard to know where to start. This is especially true when people have already spent decades trying to push the NHS in a more iterative direction, and have found their work repeatedly blocked, or dismantled when they moved on to a new post elsewhere.
So what should we do next? In part, it’s important not to feel helpless. A lot can be achieved with hustle, even within the system as it stands today. Every week I meet brilliant people who are working in and around the NHS in iterative ways already, with varying levels of permission. You can get quite a long way by applying a relentless and positive insistence, and ducking and weaving, and building support groups with allies.
On the other hand, leadership is clearly vital. Indeed if there’s one thing we learn from all of the above, it’s that senior support for these methods is even more important in healthcare than elsewhere. We need strong senior sponsors to create the conditions for Test & Learn methods. Indeed creating these conditions should be one of the primary goals for any new operating model in institutions like the Department of Health and Social Care. And the ability to create these conditions should be a standard requirement for anyone appointed into a senior leadership role, and a central focus for senior culture change programmes and leadership training.
In addition to these deeper questions of the operating environment and capabilities, I also think a big push from some top-level delivery architecture would be helpful. Partly to give Test & Learn methods the attention and senior backing they need, and partly as a cultural signal — to show that senior people (politicians and officials) recognise that these methods are vital to the delivery of the 10 Year Plan for Health.
The NHS could start by establishing 6 to 10 Test & Learn teams to work on top priorities, working with the Cabinet Office as a kind of healthcare satellite of the TLG programme. Set these teams up to work on challenges with the qualities I listed above — complex, lots of trade-offs, and so on. Pick challenges that are important and politically salient but doable — where you could make noticeable progress over, say, three years. (This is partly about building momentum, so don’t start with the most intractable challenges). Good options could include:
Home blood tests as a service (as an example of a common component or platform that could dock into a huge range of other services, visibly improving the patient experience, generating hugely valuable data, and reducing costs and time pressures across the system)
Reduce admin burden for cancer patients (as an example of a challenge that requires lots of links between systems, and that needs design-thinking, and that is very important as well as being politically salient)
Prototype a Virtual Hospital (NHS Online), or a neighbourhood health centre (as an example of building a new delivery model — showing people today what the future will look like)
Help to manage/prevent diabetes (as an example of a complex challenge that is heavily behavioural, and a huge driver of costs; an area where we need to learn how to use new technological breakthroughs (e.g. GLP1s) but also need behaviour change, and where context is important)
Support vaccinations for children, or develop a new screening programme (as an example of an area that is also behavioural, where psychology and messaging are important, and where we can use Test & Learn methods to support a big and cost-saving drive on prevention)
Build a public option for the Single Patient Record (as an example of an enabling data platform, and a strategic intervention to help break the hold of legacy private providers in an uncompetitive market)
Each of these teams would be set up to chase their outcome. The teams would be self-sufficient, including all the disciplines they need to develop and refine a live service — digital, design, policy, clinical assurance, and so on. Each team member would be empowered to make day-to-day decisions without additional sign-off, within pre-agreed guardrails. The teams would be made up of staff seconded from different parts of the NHS (including local and national; frontline and managerial), and even from across the public sector. Indeed, there would be a case for getting one or more teams to work on challenges that span different departments; an example would be ‘Help people with complex mental health conditions back into work’.
The teams would all work in the open — writing and speaking freely and regularly about their work, and sharing what they are learning, again without signing off individual communications. This is partly to surface dependencies and spread these ways of working, but also to show that steady progress is happening. The teams would have an explicit mandate to surface barriers they are hitting up against (this part doesn’t need to be public!) and these insights would flow directly to senior leaders, e.g. via regular Show and Tells with the Secretary of State, Permanent Secretary, and others.
The goal, ultimately — as with the Cabinet Office TLG programme — would be twofold. One, to make rapid progress on the 6–10 outcomes and, two, to drive changes to the system itself; in essence, to start redesigning the system bottom-up, building the system that rapid progress requires. In addition, an explicit outcome of this work would be to codify a healthcare-friendly version of Test & Learn, and an associated set of governance (e.g. revisions to the Service Standard).
Finally, In addition to these teams, the wider NHS TLG initiative could celebrate existing reformers, so that TLG isn’t being layered on as just another initiative. TLG teams would be joining a wider existing movement for change, adding new energy and permission to work that many brilliant people are already doing. To support this, the programme could also:
Find, celebrate, reward, convene, and seed fund people who are already working in iterative ways. Support demonstrations of TLG methods, wherever they originate in the system.
Invest in the relevant communities of practice and professional networks, including by investing in a proper Design, Digital, and Technology profession for health and social care, and in the system’s many other communities of reformers
Those are all just opening suggestions. There are lots of choices to make about how this work docks into the NHS and the wider government delivery architecture, and lots of debates about the best priorities to work on. The key point is that Test & Learn methods have huge potential in healthcare, but using them more widely won’t be easy, so this work needs top-level backing. And of course we can’t afford to lose time; as with planting a tree, the best time to start a Test & Learn team is yesterday.
As ever I’d be interested in feedback, critiques, and improvements. What resonates and what jars? What does this account over- or under-play? What would be the most useful work to do next? And again I should emphasise that many others contributed to this thinking, and even more people are already doing tireless work in this vein across the system.
For more on similar topics, here is a recent post on the capability needed to deliver the UK government’s 10 Year Plan for Health, and some reflections on the long fight for shorter feedback cycles in government. You can follow my writing on Blue Sky, Medium, or Substack. If you’d like to work together on these topics, drop me a line at Kinship Works — we’re an initiative supporting more adaptive, contemporary and human alternatives to bureaucracy.