This was the headline picked up by most of the media from a recent opinion poll on the issue of independence/separation, commissioned by STV from Ipsos MORI. The truth behind the headline is different and this post attempts an analysis of that number absent in most comment so far, and entirely missing from the triumphalist tone of most nationalists online.
I don’t normally devote time and effort to a detailed analysis of any single poll as the trend over time across all polls is what’s important. However, because of the praise lavished on this one by enthusiastic nationalists, not to mention the gloom it has induced in a small number of pro-GB Scots, I am happy to make an exception.
The Ipsos MORI poll comprises an unusually complex database. To check the full facts of what I’ve extracted here you may wish to go the company’s website, where (towards the bottom of the page) you will find three separate documents referenced – what they call the survey ‘topline’ giving a summary of the results; a full set of output in 29 separate ‘computer tables’; and a more user-friendly set of colourful charts. I draw on all three sources for these comments.
First, a statistical health warning that Ipsos MORI don’t spell out, except obscurely in the tiniest of footnotes to their computer tables where they flag up some results as ‘very small base (under 30) ineligible for sig testing.’ What they’re referring to here is that all properly-conducted sample surveys, and I’m sure this is one, have a margin of error associated with their estimates. Typically with a survey of about 1,000 respondents (this one has 1,029) that margin of error is about three percentage points. So straight away, the true value of an estimate of 50% could lie anywhere between 47% and 53%. That’s no comfort to either camp in what seems like our interminable debate on separation, but it is a fact.
Ipsos MORI’s pie chart at the head of this post summarises their finding. To see the detailed source of the claim that ‘50% of Scots want indy’ you have to go to Table 3 in the ‘computer tables’ (click to enlarge or check it on the Ipsos MORI website):
(The meaning of the codes in the two rows in bold, ‘bfjm’ etc is not clear)
You’ll see the 50:50 breakdown in the left-hand ‘weighted’ column. The ‘unweighted’ figures on the right show a slightly different result. For want of a better phrase weighting is a way of adjusting a particular sample to the known characteristics of an overall population. Properly done, it is quite legitimate, and it would have been helpful if the company had provided information to show how they weighted this particular sample. Without that adjustment there is a small but clear majority in favour of ‘No.’
When you look at the 50% figure you also need to be aware that it is not a figure based on the 1,029 respondents:
- it excludes people who said they didn’t know how they’d vote, or declined to say how they’d vote
- it only includes those who said in answer to another question that they were highly likely to vote in another referendum (9 or 10 out of scale of 1-10).
The bottom line is that both weighted and unweighted estimates are based on 858 people, not 1,029 (i.e. on 83% of the total sample), and any error of estimate will be slightly higher than the ±3 percentage points I mention above.
It’s also worth noting that the full question people answered was:
If a referendum were held tomorrow about Scotland’s constitutional future, how would you vote in response to the following question: Should Scotland be an independent country?
The ‘if … tomorrow’ formulation is a common one in political opinion polls but of course it’s not realistic. This poll didn’t ask another critical question – do people actually want another referendum now? An analysis of recent polls by Professor John Curtice that did ask the question suggested that only just over a third of Scots want another referendum ‘any time soon.’
The Ipsos MORI poll does have one interesting feature many similar polls lack. It not only asked the Yes/No question, it also asked people:
on a scale of 1 to 10 where 1 means ‘I completely support Scotland becoming independent’ and 10 means ‘I completely support Scotland staying part of the UK’ , what number would you consider yourself to be? (Table 9)
So a more subtle analysis of people’s strength of feelings either way is also possible. Here is Ipsos MORI’s summary table of that information, annotated by me (the brackets and summary percentages at the bottom):
Two conclusions can be drawn from this:
- the core support for staying part of the UK is significantly higher than the core support for independence – 38% vs 28%
- if you divide the whole sample into two, those tending to want independence (1-5), and those tending to want to remain part of the UK (6-10) you find a split remarkably similar to the 45:55 of 2014.
There are surely Scots on both sides of the Yes/No divide as well as those with no settled opinion who will say ‘Is that all the turmoil of the last 2½ years has produced, virtually no change from the last referendum?’ To which I would, of course, add the statutory reminder ‘once in a generation’ (©Alex Salmond/Nicola Sturgeon).
All this, and much more (see for example Nine reasons why my money’s on no more Scottish referendums …), makes me conclude that the first minister would, excuse me, be a damned fool to commit herself to another separation referendum any time in the near future.