Digging into the pollsters (GE2017)

So a few days ago I remarked on the YouGov article about how the different polling companies are all using different methods to weight their samples this time around, after the generally accepted model they all used in 2015 led them to make inaccurate predictions.

It occurred to me after the latest round of opinion polls were published that the “rolling average” graphs of voting intention being used by most media outlets – which present a mean average of the polls in the last 14-day period, or of the last 7 polls published, for example – were pretty much meaningless in this context. [CORRECTION: The BBC average uses the median average of the last seven polls, not the mean average] The results are so different from one another that they are outside of the error margin – if a poll predicts 33%, it could be anywhere from 30% to 36%. If another poll predicts 40%, then its error margin is 37% to 43%. For context, the range of results from the polls over 1st-2nd June for Labour has a high of 40% and a low of 34%, and for the Conservatives a high of 47% and a low of 40% – which is to say, they are outside of the error margins allowed.

There is no grounds for assuming that an average of the different methods will in any way be more accurate than any of the individual polls. The results aren’t comparable and the point (as outlined by that YouGov article) is to test the different methods to see which works. A rolling average tells us worse than nothing about what’s going on.

More useful would be to see individual graph lines for each of the different polling companies.

I haven’t found a source that offers this, but the Guardian’s poll tracking page does allow you to filter poll results by company, meaning you can get a crude (if annoyingly unscaled for time) graph representation of the changes in the polls and indicate what trends the polling companies indicate each in turn. (You just have to keep in mind to watch the dates of the polls and try to compare even time periods.)

Ipsos-MORI and ComRes have only published 3 polls since the election announcement, so they don’t give much of a sense of trends and changes. Most of the other companies have polled at least once a week so it’s possible to get a sense of how things are changing. Bearing in mind that the error margin is around 3%, I’ve assumed that a variation of 1-2% doesn’t mean anything but larger moves might indicate genuine changes. I’ve looked at polls taken since May’s 18 April announcement of a general election.

Panelbase shows the Tories staying level (within error margin) until their 1 June poll, where there’s a 4% drop from the previous (23 May) result of 48%. For Labour, while no individual change from week to week was outside the error margin, the trend has been consistently upwards so that overall their numbers have climbed from 27% on 24 April, to 36% in the latest.

YouGov shows something different: a gradual trend downwards for the Conservatives (with a blip on 12 May of a 3% jump up, before returning to the level predicted by the downward trend the week after). From a high of 48% on 2 May (and that off-trend result of 49% on 12 May), the Tories were on 42% in the 2 June poll. Labour, meanwhile, stayed within margin of error around 31% until a sudden jump to 38% for the 25 May poll, and subsequent polls have remained within the margin of error of that result.

TNS BMRB (who don’t seem to be included in the BBC’s results) show a mix of these two. Their results for the Tories vary up and down within margin of error, but both top and bottom of that range have trended downwards overall so that the low result of 46% for 25 April and high of 48% on 2 May, is now compared to 42% on 22 May and 43% on 30 May. Their results for Labour show two jumps outside of error margin: from a start of 24%, a jump to 28% on 8 May, and then another jump to 34% on 25 May (2 June showed 33%).

ICM poll shows Tories staying within margin of error of their 47% result on 1 May (and 22 May). The latest result on 1 June was 45%, which is within the margin of error I outlined above. Their results for Labour stay at 28% or occasional results within margin of error until their 22 May poll, which shows a big jump to 33% (their latest shows 34%, which is margin of error).

Opinium have the Tories consistent since the General Election announcement, on 46% or within error margin of it, until their 31 May result of 43%, a jump downwards. Labour trended upwards consistently from 30% on 26 April to 37% on 31 May.

ORB put the Tories on 45% in their 1 June poll and this is consistent (within a 1% error margin) of all their results since the announcement. Labour trended upwards consistently from 31% (4 May) to 36% (1 June) with a blip of 38% on 27 May.

Survation is one of the most intriguing to look at. I’m leaving out their most recent poll because the BBC reports that this used a different methodology (internet vs phone polling). For the Tories, their 22 April poll showed them on 40%, but the next poll after that showed a jump to 47%, which remained consistent until their 22 May poll, when the Tories dropped to 43%. On Labour, they show a consistent 30% level until 20 May, where they jump to 34% and remain there until a week later (27 May), they show another jump to 37%.

* * *

Out of seven companies, the division in the types of movement are split evenly: on Labour, three show a general trend upwards, while four show Labour’s result moving in occasional large jumps. On the Tories, two have them staying level, two have them trending downwards and three show a big jump downwards.

Now, judging by when the polls show big jumps, a big change happened, it seems, between 20-22 May. There was a big event on the evening of 22 May but I would absolutely not say that that was the cause. The change must have taken place before the suicide bombing in Manchester. A quick search for news around that date indicates that a possible cause was the launch of the Conservative manifesto on 18 May! Those that show a later jump for Labour, I suspect, would correlate with the Sky leaders’ “debate”.

The questions are: whether the Tories will remain steady or drop again before election day; and for Labour will the steady trend upwards continue until polling day, or are the “jump” graphs accurate, indicating that they will probably stay level from now until polling day?

There is no consensus on the shape of both graphs: the only thing I can say is that neither company that showed a Tory trend downwards also showed a Labour trend downwards: both of them indicated a sudden jump rather than a gradual change.

With less than a week until polling day, there isn’t much time for there to be further changes.

The range indicated for the Tories is 42%-45% in the most recent polls (leaving aside the recent Survation poll that put them on 40%, and that I said one of those 45% was in a margin of error range from 47%). This is quite tightly bunched and the question is, does the Survation internet poll indicate a new drop, or continuation of a downward trend, into the last week? Perhaps it indicates that the methodology is closer when it comes to modelling Tory turnout. As I noted from the YouGov article before, the big difference seems to be how they model youth turnout.

The Labour Party range is 34%-38% (again, leaving out the Survation internet poll, and also not including the Ipsos-MORI poll on 1 June which put them at 39% and 40% respectively). That looks neatly bunched with the Tories, but with Labour there is a gap in the range so that it forms two groups, a low group and a higher group, with more results in the higher group. These results have changed more over time, too, which adds a greater variability to frustrate attempts to draw conclusions.

There is still no way to use this analysis to predict the result on polling day. The behaviours are too different. But I hope it does give a better idea of what might really be going on.


About ValeryNorth

I overthink everything.
This entry was posted in Politics, Science and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s