Nov 252013

The very useful OUP blog has a Q&A session on polling accuracy with Jocelyn Evans and me: Polling accuracy: a Q&A with Kai Arzheimer and Joceyln Evans.

Moreover, Oxford University Press has kindly ungated the Political Analysis paper on our New Multinomial Accuracy Measure (TM), so grab the PDF while it is free.

Nov 222013

Measuring Survey Bias

In our recent Political Analysis paper (ungated authors’ version), Jocelyn Evans and I show how Martin, Traugott, and Kennedy’s two-party measure of survey accuracy can be extended to the multi-party case (which is slightly more relevant for comparativists and other people interested in the world outside the US). This extension leads to a series of party-specific measures of bias as well as to two scalar measures of overall survey bias.

Moreover, we demonstrate that our new measures are closely linked to the familiar multinomial logit model (just as the MTK measure is linked to the binomial logit). This demonstration is NOT an exercise in Excruciatingly Boring Algebra. Rather, it leads to a straightforward derivation of standard errors and facilitates the implementation of our methodology in standard statistical packages.

Voter poll
Those Were the / CC BY-SA

An Update to Our Free Software

We have programmed such an implementation in Stata, and it should not be too difficult to implement our methodology in R (any volunteers?). Our Stata code has been on SSC for a couple of months now but has recently been significantly updated. The new version 1.0 includes various bug fixes to the existing commands surveybias.ado and surveybiasi.ado, slightly better documentation, two toy data sets that should help you getting started with the methodology, and a new command surveybiasseries.ado.

surveybiasseries facilitates comparisons across a series of (pre-election) polls. It expects a data set in which each row corresponds to margins (predicted vote shares) from a survey. Such a dataset can quickly be constructed from published sources. Access to the original data is not required. surveybiasseries calculates the accuracy measures for each poll and stores them in a set of new variables, which can then be used as depended variable(s) in a model of poll accuracy.

Getting Started with Estimating Survey Bias

The new version of surveybias for Stata should appear be on SSC over the next couple of weeks or so (double check the version number (was 0.65, should now be 1.0) and the release date), but you can install it right now from this website:

net from 
net install surveybias

To see the new command in action, try this

use fivefrenchsurveys, replace

will load information from five pre-election polls taken during the French presidential campaign (2012) into memory. The vote shares refer to eight candidates that competed in the first round.

surveybiasseries in 1/3 , popvaria(*true) samplev(fh-other) nvar(N) gen(frenchsurveys)

will calculate our accuracy measures and their standard errors for the first three surveys over the full set of candidates.

surveybiasseries in 4/5, popvariables(fhtrue-mptrue) samplevariables(fh-mp) nvar(N) gen(threeparty)

will calculate bias with respect to the three-party vote (i.e. Hollande, Sarkozy, Le Pen) for surveys no. 4 and 5 (vote shares a automatically rescaled to unity, no recoding required). The new variable names start with “frenchsurveys” and “threeparty” and should be otherwise self-explanatory (i.e. threepartybw is $B_w$ for the three party case, and threepartysebw the corresponding standard error). Feel free to plot and model to your heart’s content.

Jul 172013

Replication data for our forthcoming Political Analysis paper on our new, multinomial accuracy measure for bias in opinion surveys (e.g. pre-election polls) has just gone online at the PA dataverse. So if you want to gauge the performance of French pollsters over the 2012 presidential campaign (or cannot wait to re-run our simulations of B_w‘s sampling distribution), download our data and start playing.

A current version of our Stata add-on surveybias is included in the bundle. Alternatively, you can install the software into you personal ado dir by typing ssc install surveybias.

Jul 102013
Used Punchcard
BinaryApe / Foter / CC BY

In a recent paper, we derive various multinomial measures of bias in public opinion surveys (e.g. pre-election polls). Put differently, with our methodology, you may calculate a scalar measure of survey bias in multi-party elections.

Thanks to Kit Baum over at Boston College, our Stata add-on surveybias.ado is now available from Statistical Software Components (SSC).  The add-on takes as its argument the name of a categorical variable and said variable’s true distribution in the population. For what it’s worth, the program tries to be smart: surveybias vote, popvalues(900000 1200000 1800000), surveybias vote, popvalues(0.2307692 0.3076923 0.4615385), and surveybias vote, popvalues(23.07692 30.76923 46.15385) should all give the same result.

If you don’t have access to the raw data but want to assess survey bias evident in published figures, there is surveybiasi, an “immediate” command that lets you do stuff like this:  surveybiasi , popvalues(30 40 30) samplevalues(40 40 20) n(1000). Again, you may specify absolute values, relative frequencies, or percentages.

If you want to go ahead and measure survey bias, install surveybias.ado and surveybiasi.ado on your computer by typing ssc install surveybias in your net-aware copy of Stata. And if you use and like our software, please cite our forthcoming Political Analysis paper on the New Multinomial Accuracy Measure for Polling Bias.

Update April 2014: New version 1.1 available

Dec 022011

Who is afraid of whom?

The liberal German weekly Zeit has commissioned a YouGov poll which demonstrates that Germans are more afraid of right-wing terrorists than of Islamist terrorists. The question read “What is, in your opinion, the biggest terrorist threat in Germany?” On offer were right-wingers (41 per cent), Islamists (36.6 per cent), left-wingers (5.6 per cent), other groups (3.8 per cent), or (my favourite) “no threat” (13 per cent). This is a pretty daft question anyway. Given the news coverage of the Neo-Nazi gang that has killed at least ten people more or less under the eyes of the authorities, and given that the authorities have so far managed to stop would-be terrorists in their tracks, the result is hardly surprising.

Nonetheless, the difference of just under five percentage points made the headlines, because there is a subtext for Zeit readers: Germans are worried about right-wing terrorism (a few weeks ago many people would have denied that there are right-wing terrorists operating in Germany), which must be a good thing, and they are less concerned about Islamist terrorists, which is possibly a progressive thing. Or something along those lines.

But is the five-point difference real?

YouGov has interviewed 1043 members of its online access panel. If we assume (and this is a heroic assumption) that these respondents can be treated like a simple random sample, what are the confidence intervals?

Binomial Confidence Intervals

First, we could treat the two categories as if they were distributed as binomial and ask Stata for exact confidence intervals.

cii 1043 round(1043*.41)
cii 1043 round(1043*.366)

The confidence intervals overlap, so we’re lead to think that the proportions in the population are not necessarily different. But the two categories are not independent, because the “not right-wingers” answers include the “Islamists” answers and vice versa, so the multinomial is a better choice.

Multinomial Model

It is easy to re-create the univariate distribution of answers in Stata:

set obs 5
gen threat = _n
lab def threat 1 "right-wingers" 2 "islamists" 3 "left-wingers" 4 "other" 5 "no threat"
lab val threat threat

gen number = round(1043* 0.41) in 1
replace number = round(1043* 0.366) in 2
replace number = round(1043* 0.056) in 3
replace number = round(1043* 0.038) in 4
replace number = round(1043* 0.13) in 5
expand number

Next, run an empty multinomial logit model

mlogit threat,base(5)

The parameters of the model reproduce the observed distribution exactly and are therefore not very interesting, but the estimates of their standard errors are available for testing hypotheses:

test [right_wingers]_cons = [islamists]_cons

At the conventional level of 0.05, we cannot reject the null hypothesis that both proportions are equal in the population, i.e. we cannot tell if Germans are really more worried about one of the two groups.


Just for the fun of it, we can carry out one additional test and ask a rather specific question: If both proportions are 0.388 in the population and the other three are identical to their values in the sample, what is the probability of observing a difference of at least 4.4 points in favour of right-wingers?

The idea is to sample repeatedly from a multinomial with known probabilities. This could be done more elegantly by defining a program and using Stata’s simulate command, but if your machine has enough memory, it is just as easy and possibly faster to use two loops to generate/analyse the required number of variables (one per simulation) and to fill them all in one go with three lines of mata code. Depending on the number of trials, you may have to adjust maxvars

local trials = 10000
foreach v of newlist s1-s`trials' {
qui gen `v' = .

probs =(.388,.388,.056,.038,.13)
X[.,.] = rdiscrete(1043,`trials',probs)

local excess = 0

forvalues sample = 1/`trials' {
qui tab s`sample' if s`sample' == 1
local rw = r(N)
qui tab s`sample' if s`sample' == 2
local isl = r(N)
if (`rw' / 1043 * 100) - (`isl' / 1043 * 100) >=4.4 local excess = `excess' +1

display "Difference >=4.4 in `excess' of `trials' samples"

Seems the chance of a 4.4 point difference is between 5 and 6 per cent. This probability is somewhat smaller than the one from the multinomial model because the null hypothesis is more specific, but still not statistically significant. And the Zeit does not even have a proper random sample, so there is no scientific evidence for the claim that Germans are more afraid of right-wing extremists than of Islamists, what ever that would have been worth. Bummer.

Apr 092011

Sometimes, a man’s gotta do what a man’s gotta do. Which, in my case, might be a little simulation of a random process involving an unordered categorical variable. In R, sampling from a multinomial distribution is trivial.


gives me a vector of random numbers from a multinomial distribution with outcomes 1, 2, 3, and 4, where the probability of observing a ‘1’ is 10 percent, the probability of observing a ‘2’ is 70 per cent, and so on. But I could not find an equivalent function in Stata. Generating artificial data in R is not very elegant, so I kept digging and found a solution in section M-5 of the Mata handbook. Hidden in the entry on runiform is a reference to rdiscrete(r,c,p), a Mata function which generates a r*c matrix of draws from a multinomial distribution defined by a vector p of probabilities.

That leaves but one question: Is wrapping a handful of lines around a Mata call to replace a non-existent Stata function more elegant than calling an external program?