The good and weak pollsters in Macedonia

The name issue, perception of state institutions, their trustworthiness, ratings of political parties and other political and social actors, corruption, importance of inter-ethnic relations, euroatlantic integrations and so on. The public mood about these and other issues are the best indicators of the public will, that should be the guideline of action for any government in Macedonia. Unfortunately these issues are not easily quantifiable or that can be calculated by a formula in a direct and precise manner, and so the burden of measuring them falls fully on pollsters. But the polling process itself is never one hundred percent accurate, which leaves us with the dilemma of how to really read their results. On top of that pollsters in Macedonia have a long history of wrong predictions or straight-out manipulations for propaganda purposes that leave big question marks about their reliability.

Of course not all pollsters err the same or in the same direction. How then to recognize the reliable predictions and separate the good pollsters from the weak ones? The only method is to compare them against a quantity that can be otherwise measured directly and accurately, and that quantity is the ratings of political parties that are regularly measured in elections. Starting with the last parliamentary elections, we have compared their results with the predictions of pollsters during the last month before elections. By measuring the deviation we can then construct a table that shows who exactly were the good pollsters and who the weak ones.

The first party in 2016 elections was VMRO which got 38% of all valid ballots, followed by SDSM with 36.8%. Third was DUI with 7.3%, then Besa with 4.9%, Alliance with 3% and DPA with 2.6%.

There were six pollsters who were active before elections. For each of them we took into account their last poll. Their accuracy is then measured through the average of errors for each party (through the “root of squares”). Their ranking then is as follows: the most accurate was Agency Rejting with an average error of 2.1%, then TV Telma with M-Prospekt with 7.0%, Faktor Plus Adria 8.3%, Brima-Gallup 11.3%, Pavel Shatev 13.7% and the most innacurate was Dimitrija Cuposki with an error of 15.1%.

This table makes it clear that the only reliable pollster is Agency Rejting which is far ahead of the rest of the pack. Telma and Faktor Adria Plus are at best average pollsters, while the remaining three are weak and may be easily dismissed.

However that is not the full picture. As the reader might guess the error with Pavel Shatev and Dimitrija Cuposki comes entirely by underestimating SDSM and overestimating VMRO, with a total 16% difference from reality. Faktor Adria and Telma also underestimated SDSM but only by a net 6%, while by far the most accurate was Rejting that erred by only 1.4% in the ratio VMRO-SDSM. This means that when reading VMRO-SDSM head to head, the only reliable pollster is Rejting; Faktor Adria and Telma should be read with caution, while the rest may be completely ignored.

It is an entirely different story with Albanian parties. Rejting is first again, but here it is followed by Pavel Shatev and Dimitrija Cuposki. Faktor Adria and Telma are far weaker, while Brima-Gallup errs in the same levels as with the Macedonian bloc of parties. In general we see an overestimation of DUI and DPA, and underestimation of the new parties Besa and Alliance.

Especially disappointing for the Albanian bloc of parties are the predictions of TV Telma in cooperation with M Prospect. This pollster did not even manage to guess the correct ranking, which should be an especially easy task for any pundit. The pollster predicted that DPA would be the second party in elections, far ahead of Besa and Alliance. On top of this it predicted the results for each electoral district separately, giving 14 MPs to DUI, 5 each for Besa and DPA and only one for the Alliance. For any pollster this should have been a wake-up call and a driver to change its polling model, but unfortunately it continued to predict similar results even after elections, which were again proved wrong during local elections.

Obviously polling itself is no easy task, especially so in Macedonia where there is still a lingering fear of repercussions if one speaks publicly about their political preferences. However the fact that Rejting came very close to the correct results shows that it is not impossible. It gives hope that the other pollsters, or new ones that might appear, will improve their models, of course given that they do not have a preset political agenda.

As things stand now the results of all pollsters except Rejting should be read with a healthy dose of caution. One should be especially careful with predictions of Pavel Shatev and Dimitrija Cupovski in the Macedonian bloc, and Telma and M Prospekt when they predict results in the Albanian bloc.

Robert Nesimi