Anyone who follows this blog will know that I occasionally comment on polls and their significance. I don’t take an obsessive interest in the detail. I always caution that the trends are important, not individual polls, and that the errors associated with sample size need to be taken into account. Nationalists often think I’m putting a spin on what I say, but nearly all polls reveal facts that have been, until now, favourable to those of us who want to maintain the constitutional status quo.
In the last couple of days two ‘polls’ (you’ll realise shortly why I put that word in inverted commas) have had many separatists jumping up and down with joy. One purported to show that 51% of its particular sample was in favour of independence, the other a massive 57%. If they’re true, they represent a major and sudden reversal in the long term trend that has shown modest but slowly increasing majorities for ‘No.’
The trouble is, neither figure can be trusted. Here’s why.
The first poll was highlighted on the front page of The National newspaper:
The full text of the article said:
yesterday a poll revealed 51 per cent of voters were now in favour [of independence]. It found 41 per cent of voters wanted Scotland to be an independent member of the European Union, while a further 10 per cent favoured independence outside the bloc.
It stated that the figures were taken from a Panelbase poll commissioned by the Sunday Times.
Similar figures had been reported earlier on Twitter by an account called @EuropeElects (‘Psephology in the European Union’):
which also had the same data on its Facebook page. If you check, you’ll see that all they do is post a stream of summaries of polls and elections from across the EU without any accompanying commentary. It’s pretty indigestible, but to see the Facebook version of this data, scroll down to their post on 30 April at 20:15, together with the comment it’s attracted from other people.
The problem in all this detail is that the full results of the Panelbase survey, on their website, actually ask the direct question:
And how would you vote in response to the question: Should Scotland be an independent country? (Likely voters)
and the answers are shown:
That is, using the usual shorthand, 52% said No and 43% said Yes or, extracting the Don’t Know’s, 55% No, 45% Yes, which will sound familiar to all Scots who feel stuck in ground hog day since September 2014.
As far as I can work out, The National’s error (and EuropeElects) comes from adding two separate figures from elsewhere in the poll results for those that want independence in the EU and outwith the EU. I can’t see those results in the Panelbase tables linked above. So they are either in unpublished tables and/or appear behind the Sunday Times paywall.
For the record, EuropeElects subsequently publicised the correct data:
although without any explanation or apology.
The second ‘poll’, and here the inverted commas really are necessary, was publicised on the Daily Record’s website under the headline
Poll shows most Scots would prefer independence in Europe rather than face Tory rule in UK after Brexit
And the eye-catching graphic:
This has led innumerable enthusiasts to tweet ‘Support for indy up to 57%!’ (or similar).
However, again, it needs a detailed look to understand what this exercise really is.
If you scroll down to the bottom of the Record article, you’ll find that the statistics were compiled from a ‘Google survey’:
Google Surveys run thousands of surveys a day, across a network of online news, reference and entertainment sites where it’s embedded directly into content.
On the web, users answer questions in exchange for access to that content. The user’s gender, age, and geographic location are inferred based on anonymous browsing history and IP address.
Using this data, Google Surveys can automatically build a representative sample of thousands of respondents.
You might recognise something familiar struggling to emerge from this jargon. I encountered it when I went online to look at the Record’s article, as you will. Before you can view the article you have to answer a question, in my case which of four retailers’ online videos I’d seen.
Let me suspend my disbelief for a moment about this particular exercise and acquaint you with the criteria I use to judge whether a survey is reputable and reliable.
- Is it conducted by a professional market research/polling organisation?
- Is it clear what the ‘population’ for the survey is, e.g. all adults of voting age in Scotland?
- Has a sample been chosen from the population either randomly (preferably) or adjusted to be representative of the population overall?
- Is the sample size large enough to be reasonably sure that it could be representative?
- Is it clear who the survey was carried out for, when, and how?
- Have the results been published in a systematic way e.g. in a series of tables?
- Is advice available on how reliable the results might be e.g. a ‘statistical error of estimate’ that says something like ‘Figures in this survey should be reliable with ± 3 percentage points’?
- Has a full research report been published where I can see all this information?
If I then test the Daily Record/Google survey against these criteria it does not meet one of them, with the arguable exception of being carried out by a ‘professional market research/polling organisation.’ Even there I have my doubts for the purpose of political opinion polling, notwithstanding the claim that Google Surveys carry out ‘thousands’ of surveys every day.
The Record article claims ‘There were a lot of mixed messages in the polling data with politics in Scotland in a state of flux.’ On the question of independence/separation I would argue on the contrary that the situation is remarkably, and for all sides, frustratingly stable. Any messages from the so-called ‘polling’ data here are likely to be mixed because, for the reasons I’ve already outlined, the whole exercise is of very little statistical value. Anyone, of whatever persuasion, who draws any conclusions from it is clutching at straws.
In particular, nationalists (and it would only be propagandists or the naïve) who take comfort from either of these exercises are likely to be deeply disappointed.
Regular readers will know that I prefer to look at the trend in polls, so by way of conclusion I repeat updated versions of two graphs of reputable polls I have used before on this blog. I leave you to draw your own conclusions from them.
What Scotland thinks, Yes (blue), No (orange) 25 June 2016 – 27 April 2017
rwbblog Yes (yellow), No (blue) October 2014 – April 2017
(Using the ‘best fit’ from a statistical technique called regression analysis, this shows a long term trend since the independence referendum towards ‘No’)