Yes Prime Minister, questionnaire design matters

In a famous Yes, Prime Minister episode Sir Humphrey Appleby once explained to Bernard Woolley how you could get contradictory polling results on the same topic – in this case the reintroduction of national service – by asking a series of leading questions beforehand and asking the key question you want to know about in a certain way. The clip is here.

But what would happen if we asked Sir Humphrey's questions today? To find out we asked 1,000 British adults the first set of questions that were positive about national service and 1,000 British adults the second set that were negative. Below is a comparison of the results. You can see that it is indeed true that you get different results on the level of support for the reintroduction of national service based on the way you ask the question and the questions you ask before it.

So what does this tell us?

First of all, to be clear, we would never ask questions on such a topic in this way. These are taken from a comedy sketch, a sketch that works because of the obvious absurdity of what Sir Humphrey is suggesting.   At Ipsos, we take great care not to ask leading questions. Questionnaire design is a key part in how our researchers are trained. Our professional reputation is based on providing quality data and acting with integrity. It is why clients come to us.

Related to this, it is worth pointing out that Sir Humphrey's central allegation – that polling companies don’t publish all of the questions they include in surveys – is not allowed under British Polling Council rules. Under these rules, polling companies have to be transparent. They must publish all relevant questions asked in a poll in the order they were asked. Interested parties can then reasonably disagree on question wording and so on in an open way. In many ways this acts as something of a public peer review of survey results.

All of this matters in a General Election year. There will be lots of polling between now and the election, perhaps more than ever before and polling around policy choices will be particularly important when the various parties’ manifestos are launched. It is always worth paying close attention to how questions are asked and looking at a range of polls from different places, not just one, when forming a view on what the public think. Finally, it is worth watching the trend over time too. How the public feel about a particular policy might change depending on how it gets presented in the media and the arguments that are made for and against. As Sir Humphrey's questions point out in their own way, the way a policy choice is framed matters in whether the public will support it or not. Public opinion is rarely set in stone.

Technical note

Ipsos interviewed a representative quota sample of 2,158 adults aged 16-75 in Great Britain. Half saw the ‘Sample A’ questions, reflecting a positive view about national service. Half saw ‘Sample B’, reflecting a negative view. Interviews took place on the online Omnibus 2-4 February 2024. Data has been weighted to the known offline population proportions. All polls are subject to a wide range of potential sources of error.