Note: This post was originally published by the Institute for Government on 25 June 2015 and is re-posted here with permission.
When budgets are tight, governments tend to treat experimentation and evaluation as something of a luxury. But it’s never been more important to establish what works and ensure that spending decisions are informed by demonstrable results. So what chance is there that experimentation and evaluation will be spared the chopping block the Autumn Spending Review? Jen Gold looks at the importance of a new cross-government support service aimed at helping departments run more experimental trials.
Tuesday [23 June] saw the What Works Team in the Cabinet Office launch the government’s Trial Advice Panel – a free service developed in partnership with the Economic and Social Research Council that offers Whitehall departments technical support in designing and implementing controlled experiments. This innovation owes much to the work of the Behavioural Insights Team (BIT).
Set up in the Cabinet Office in 2010, BIT has promoted the application of insights from the behavioural sciences in the design of public policy and done so through a steadfast commitment to supporting departments in finding out what works through using randomised control trials (RCTs) and other types of quasi-experimental methodologies. BIT’s much-downloaded report of 2012, Test, Learn, Adapt, made a persuasive case for expanding the use of controlled experiments in government and David Halpern, BIT’s Chief Executive, went on to be appointed the UK Government’s National What Works Adviser.
At a recent roundtable we held with NESTA’s Alliance for Useful Evidence to mark the publication of their report on experimental government, it was clear that we have seen increased experimentation in Whitehall over the last few years. But experimental and quasi-experimental methodologies are still under-used.
experimental and quasi-experimental methodologies are still under-used
Of course, there are plenty of demand-side causes for this lack of experimentation (as highlighted in our report on the use of evidence and evaluation in policymaking). But there is also a significant capability gap. It’s this lack of widespread expertise in experimental methods that the Trial Advice Panel – set up by Halpern’s What Works Team – is designed to address.
Made up of 25 academics and a cross-government group of trialling experts, the panel offers departments a range of free services, including ongoing technical support to a select number of trials via small sub-committees of panel members; an email service where individual panel members can field specific technical questions posed by departments; and, depending on demand, drop-in “surgeries” within departments. Advice from the panel will go beyond randomised controlled trials to include other types of controlled experiment such as propensity score matching.
All of this represents a smarter way of sharing in-house expertise – acknowledging recent advances made by departments, including:
- the Ministry of Justice, who have developed considerable expertise in quasi-experimental methodologies, most notably propensity score matching;
- the Department for Work and Pensions, who are running large-scale randomised controlled trials with the support of an internal trials unit;
- the Department for International Development, who have heavily invested in trials and evaluation projects over the past five years.
The panel also offers a means of tapping into the skill sets of academics that have first-hand experience of running high-quality trials. By partnering with the Economic and Social Research Council (ESRC), the Cabinet Office has made use of the Council’s convening power. The carefully selected group of academics come from a range of disciplinary backgrounds relevant to current trialling needs in government including, public health, clinical medicine, economics, statistics, education, international development, and crime and justice.
This type of “knowledge exchange”, as Jane Elliott (Chief Executive of the ESRC) pointed out at the launch event, is more important than ever now that funding is constrained. And with so many Panel members having extensive experience of low-cost, rapid testing methodologies, the initiative has the potential to help departments overcome one of the biggest obstacles to testing: the pressure to make quick decisions under tight budgets before any results from potential trials would ever be known.
For the panel to succeed in increasing the supply of evidence that informs policy making, it is critical there is sufficient demand at the right stage in the policy design process. At Tuesday’s launch, Chris Wormald, Head of the Civil Service Policy Profession, highlighted the continued need for both officials and policy professionals to understand the potential of empirical methods.
…a continued need for both officials and policy professionals to understand the potential of empirical methods
If that happens, the Cabinet Office, who should be commended for arriving at a smarter way of making use of existing capability, will be able to claim credit. The experience of the Behavioural Insights Team suggests that it may only take a few success stories from supported trials to stimulate considerable demand. As David Halpern speculated at the launch event, enabling more and more effective experimentation may well prove BIT’s greatest legacy.
Jen Gold is a Senior Researcher at the Institute for Government, an independent charitable think tank that works to improve the effectiveness of UK government. For more information visit www.instituteforgovernment.org.uk