What do MPs think of randomised controlled trials (RCTs)?

A survey of MPs' attitudes has found unexpected support for using randomised controlled trials to test social policy. It also found tensions over fairness, and a preference for personal stories when talking to the public.

What do MPs think of randomised controlled trials (RCTs)?

A survey of MPs’ attitudes has found unexpected support for using randomised controlled trials to test social policy. It also found tensions over fairness, and a preference for personal stories when talking to the public.

Today’s report, from a face-to-face survey of MPs conducted by Ipsos MORI for the charity Sense About Science1, shows that MPs:

  • Support the use of controlled trials to design and test social policy
    • 67% either tend to agree or strongly agree.
  • Expect that the use of trials in policy will increase.
    • 40% agree that more trials of social policy in the next few years are inevitable, only 22% disagree.
  • Don’t consider the cost of running policy trials a barrier
    • Fewer than one-in-ten (9%) agreed that “controlled trials are too expensive as ways of designing and testing social policies”, whereas 47% of MPs, from all parties, disagreed with this statement.
  • Show some confusion about the purpose of control groups
    • 35% think that randomly choosing who gets a policy intervention is unfair. Even 26% of MPs who support the use of trials agreed with this statement.
    • 64% were in favour of using pilot schemes without control groups, though this might show that they considered them better than nothing.
    • The absence of strong views on most questions may indicate weak understanding and engagement with the benefits of policy trials.
  • Strongly emphasise anecdote and personal experience to justify policies to voters
    • More than two-thirds of MPs questioned say they’ve used personal experiences (70%) and their own principles (73%) to justify policies, but very few (8% and 34% respectively) believed these to be among the things that politicians should pay most attention to. In fact, taken together these responses suggest sensitivity to expertise and evidence, and to the human terms in which social issues and decisions need to be communicated. (The public’s preference for research to be communicated with vivid stories emerged in a study in 2014 by Ipsos MORI and the Royal Statistical Society.2)

Prateek Buch, policy director of the ‘Evidence Matters’ campaign at Sense About Science, said:

“One of the most surprising findings, during an election fought on priorities for state spending, is that few MPs object to the costs of running randomised controlled trials. We hope it is because MPs expect more effective policies to emerge from them. Of course it will only be maintained with good judgement on where RCTs would be most useful, but such broad support for policy experiments will be welcomed by the many bodies who have been pressing for better use of evidence in policy, in areas from nutrition to education to prison reform. Sense About Science has concluded that advocates of evidence in policy now need to stop moaning about MPs, appreciate their skill at communicating, and work with MPs to help them develop a better understanding of randomised controlled trials. Following the election on 7th May there may be fewer MPs with a background in scientific research, statistics or medicine especially as some champions of science such as Andrew Miller and David Willetts are stepping down. While it is not necessary to have a background in research to interrogate policies or to press for sound evidence it does make the task of improving the understanding of randomised controlled trials among MPs even more important.”

Tracey Brown, Director of Sense About Science, said:

“It’s clear that the research community needs to explain the benefits of randomised controls more clearly. They are the closest thing a politician will ever have to a magic wand, but not easy to explain to the public. MPs are adept at discussing difficult subjects in human terms, something their critics might forget. So it’s important that they understand well why randomisation is so effective and can communicate this to constituents when it might seem ‘unfair’. We will now involve them in developing our Ask for Evidence public help centre,7 but would like to see research bodies take up the challenge too.

Ben Goldacre, doctor and writer, said:

"It's great that so many MPs are enthusiastic about getting good evidence on what works in government, and we need to get better at helping them to learn more about how RCTs work. There are some very worrying knowledge gaps here. Many MPs say they're worried that RCTs are "unfair", because people are chosen at random to receive a new policy intervention: but this is exactly what already happens with "pilot studies", which have the added disadvantage of failing to produce good quality evidence on what works, and what does harm."

Bobby Duffy, Managing Director of the Ipsos MORI Social Research Institute, said:

“We have seen rapidly growing discussion of the use of randomised trials in social policy in recent years and we were keen to explore this in our MPs survey, the first time such questions have been put to MPs. Looking across the survey as a whole, it seems around 10% of MPs are firmly in favour of trials, 10% firmly against – with the large majority in the middle, but on balance seemingly open to their benefits. So there is a lot to play for - which makes a proper discussion of the advantages of trials even more important.”

Notes

1. Sense About Science (www.senseaboutscience.org) is a charity that helps people to make sense of science and evidence and promotes use of evidence in public life. This takes us from responding to outlandish diet claims by celebrities to helping parents understand vaccines, from pressing for drugs companies to publish all trials on medicines to pressing for sound use of statistics in media reporting.

2. Perils of perception, a survey conducted by Ipsos MORI for the Royal Statistical Society, found that public opinion is often based on misperceptions which run counter to evidence and is heavily influenced by memorable anecdotes (/researchpublications/researcharchive/3188/Perceptions-are-not-reality-the-top-10-we-get-wrong.aspx).

Ask for Evidence (www.askforevidence.org) is a public campaign that helps people request for themselves the evidence behind news stories, marketing claims and policies. AskforEvidence.org was developed thanks to the support of the Wellcome Trust, the Esmée Fairbairn Foundation and hundreds of individual supporters.

3. Ask for Evidence Help Centre (www.askforevidence.org/help) helps people make sense of evidence they are sent after making an Ask for Evidence request. The help centre gathers together expertise from Sense About Science and our partner organisations, shaped by our experiences and insights from 12 years helping people understand evidence and answering public questions about science. Members of the public are also able to request help directly from an expert on specific issues.

Homework for MPs: MPs who want to learn more about RCTs in public policy should watch this short video by Ben Goldacre and read the paper he wrote with civil servants, called "Test, Learn, Adapt".

Technical Note

  • Ipsos MORI carried out face-to-face interviews with 107 MPs from 4 November to 19 December 2014
  • An initial sample of 343 MPs were contacted to take part
  • Data are weighted to reflect the true balance of the House of Commons by political party and ministerial or spokesperson position
  • The following preamble preceded all the questions shown here: As you know, there are many ways of testing the effectiveness of social policies, in areas such as education, crime, health and welfare. For example, in a controlled experiment or trial, some people are randomly chosen to get a policy intervention and others do not get it at all. These groups are then compared to see the effect the policy has had. On the other hand, a pilot scheme is when a policy is tested with part of a population before being rolled out to the whole target population.
  • More information on the Ipsos MORI MPs surveys

More insights about Science