This is the author’s version of the work. Please cite as:

Arzheimer, Kai and Jocelyn Evans. “A New Multinomial Accuracy Measure for Polling Bias.” Political Analysis 22.1 (2014): 31–44. doi:10.1093/pan/mpt012
[BibTeX] [Abstract] [Download PDF] [HTML] [DATA]

In this article, we propose a polling accuracy measure for multi-party elections based on a generalization of Martin, Traugott, and Kennedy s two-party predictive accuracy index. Treating polls as random samples of a voting population, we first estimate an intercept only multinomial logit model to provide proportionate odds measures of each party s share of the vote, and thereby both unweighted and weighted averages of these values as a summary index for poll accuracy. We then propose measures for significance testing, and run a series of simulations to assess possible bias from the resulting folded normal distribution across different sample sizes, finding that bias is small even for polls with small samples. We apply our measure to the 2012 French presidential election polls to demonstrate its applicability in tracking overall polling performance across time and polling organizations. Finally, we demonstrate the practical value of our measure by using it as a dependent variable in an explanatory model of polling accuracy, testing the different possible sources of bias in the French data.

@Article{arzheimer-evans-2013,
author = {Arzheimer, Kai and Evans, Jocelyn},
title = {A New Multinomial Accuracy Measure for Polling Bias },
journal = {Political Analysis},
year = 2014,
abstract = {In this article, we propose a polling accuracy measure for
multi-party elections based on a generalization of Martin,
Traugott, and Kennedy s two-party predictive accuracy index.
Treating polls as random samples of a voting population, we first
estimate an intercept only multinomial logit model to provide
proportionate odds measures of each party s share of the vote, and
thereby both unweighted and weighted averages of these values as a
summary index for poll accuracy. We then propose measures for
significance testing, and run a series of simulations to assess
possible bias from the resulting folded normal distribution across
different sample sizes, finding that bias is small even for polls
with small samples. We apply our measure to the 2012 French
presidential election polls to demonstrate its applicability in
tracking overall polling performance across time and polling
organizations. Finally, we demonstrate the practical value of our
measure by using it as a dependent variable in an explanatory model
of polling accuracy, testing the different possible sources of bias
in the French data.},
keywords = {meth-e},
volume = {22},
number = {1},
pages = {31--44},
doi = {10.1093/pan/mpt012},
url = {https://www.kai-arzheimer.com/new-multinomial-measure-polling-bias.pdf},
data = {https://doi.org/10.7910/DVN/1V0FCS},
html =
{https://www.kai-arzheimer.com/new-multinomial-accuracy-measure-for-polling-bias}
}

This is the ungated final version. Click here for an ungated PDF of “A New Multinomial Accuracy Measure for Polling Bias”.

To install our Stata add-on, type ssc install surveybias or click here.

For replication scripts and data, click on “Data”.

A New Multinomial Accuracy Measure for Polling Bias

1 Introduction

Work on pre-election polls forms a vital part of election analysis and forecasting in both academic publications and media coverage. Frederick Mosteller (Mosteller et al., 1949) introduced the notion of accuracy measures to assess polls against actual election results. Those indices are designed for two-party/-candidate races, and cannot easily be applied to multi-party/-candidate elections. An equivalent index has not so far been derived for multi-party elections, limiting the ability of researchers to measure overall polling accuracy in such cases.

Starting from the predictive accuracy measure proposed by Martin, Traugott and Kennedy (2005), we propose such an accuracy measure, B, for elections with more than two parties or candidates. First, we derive this index mathematically from an implementation of the multinomial logistic model, including the relevant tests of statistical significance. We then consider how this aggregate measure may be biased, given it is based upon compositional data, and use a simulation to examine the extent of this bias. Finally, we use the B measure as the dependent variable in an explanatory model of different sources of polling bias, as an illustration of how the measure may be applied empirically.

The rest of this paper is rather math-heavy and does not display well as HTML. Go here for the PDF version of the preprint.

The final version of this paper should appear later this year in Political Analysis. We have created a Stata command surveybias that implements the methods described in the paper. It is due to appear on SSC this summer.