Reporting the Polls - a Lot of Hot Ayr

"POLL SHOCK: VOTERS TO GIVE DEWAR A BLOODY NOSE. LABOUR FACE AYR CRASH - EXCLUSIVE BY RON MACKENNA" screams the front page of yesterday's Daily Record, Scotland's highest circulation daily newspaper. It certainly shocked me - but it was the accuracy of the reporting, not the data in the poll (a constituency poll by Scottish Opinion Limited ahead of next week's Ayr by-election) that was disturbing.

To begin at the beginning: the voting intention figures in Ayr are reported as percentages, as is normal, but they are percentages of the entire sample, including don't knows, which is not. It has been the invariable practice of British pollsters to report voting intention figures "re-percentaged", i.e. as a percentage of those naming a party or candidate, excluding the don't knows and won't says. This enables the figures to be compared directly with previous election results, for swing to be calculated, and when the time comes for the pre-election polls to be compared with the final result. (For instance, the Tories had 34% of the Ayr vote in the 1997 general election, 38% in the 1999 Scottish Parliament election, and have approximately 41% of those expressing a preference in the Record poll.) While it is not, strictly speaking, wrong to present the figures without re-percentaging, the convention is so universal that it can only cause confusion to readers.

Of course, the proportion of don't knows should be reported as well, and are sometimes (as in this case) a legitimate story in their own right. When MORI's voting intention figures are reported in The Times, or indeed on this web site [February 2000 Times poll], the technical note invariably reports the percentage of the original sample excluded from the voting figures - split into three categories, those who specifically state that they would not vote, those who are undecided and those (usually only a very few) who refuse to state how they would vote.

The American Primary Election season, now in full swing, offers a reminder of just how confusing it can get when not all the poll figures are calculated on the same basis. Confusing to everyone - journalists, politicians, even pollsters, but especially poor old Joe Public, trying to make senses of the whole thing on the basis of media reports. Over there, there are polls based on all adults, on all registered voters or registered party supporters, on all those likely to vote, and all of these may or may not exclude don't knows; you could get five or six different figures from exactly the same sample of the public, depending on how you chose to do the calculations - and it seems that almost every possible method of calculation is being used by at least one of the pollsters. No two polls mean the same thing, and one poll can rarely be compared with another without recalculating the figures.

But to return to the Record's "Exclusive".

The poll "shows Labour have now been overhauled by the SNP...", states the Record. Not so. It shows the SNP with the support of 18% and Labour with 17%, based on a sample of 1,010. The 'margin of error' (technically, the 95% confidence limit) for a measurement of 18% in a poll with this sample size is around 1772.4 percentage points. Statistically, the survey does not show either that the SNP is ahead of Labour or vice-versa; the contest between them is too close to call. It is, fractionally, more likely that the SNP is ahead than that Labour is; that is all.

Worse, it goes on to claim that "[Labour] seem certain to finish up in third place next week". But even taking the poll at face value, the SNP's lead over Labour amounts to 1% of the electorate, yet 37% are still undecided and, of those, eleven times as many say they voted Labour last time as voted for the SNP! Not the sort of certainty you would want to bet your house on.

But it gets worse. The graphic sets out the percentages of the don't knows who voted for each of the four main parties in last year's election - 77% Labour, 11% Conservative, 7% SNP and 4% Liberal Democrat. But Ron Mackenna's article has some of these figures the other way round: instead of saying that 11% of don't knows are ex-Conservatives and so on, it reports that 11% of ex-Conservatives are don't knows - "all but 11 per cent of recent Tory voters have made their minds up as have all bar seven per cent of the SNP's".

This is not good reporting. Furthermore, it will no doubt get worse in the game of Chinese whispers that tends to take the place of secondary reporting of poll stories. Other papers, and broadcasters, will probably repeat the figures out of context, misinterpret them, perhaps create further errors ... and in a week's time when the votes have been counted, somebody will be telling us "the polls got it wrong again, they said Labour would only get 17%".

Must do better - please.