Any rigorous discussion of survey bias requires a scalar measure of survey accuracy
For metric variables, this is trivial (e.g. deviation from the known mean in the population). But for categorical variables such as vote choices, there is no obvious solution.
Martin, Traugott, and Kennedy (2005) propose a measure for quantifying survey bias in the dichotomous (two-party) case. Arzheimer and Evans (2014) demonstrate how measures for the more common multi-party case can be derived by flagging up the relationship of the original Martin, Traugott, and Kennedy measure to the familiar logit model and then moving on from the binomial to the multinomial logit. The are category-specific measures of survey bias, whereas neatly summarise survey bias (with respect to one categorical variable) in a single figure. is a weighted variant of that takes the (true) size of the respective groups into account.
The math behind these measures is not very complex but a bit convoluted. You could do the calculations by hand (shudder) or with a bit of programming in some statistics/math package, but that would be tiresome. So we have created an add-on (or rather and ado) for Stata does the gruntwork for you.
The latest version (1.4) of the software has many new features and works much faster than previous releases.
The surveybias Software for Assessing Bias in Surveys
To install the latest version of our software from within Stata, simply type:
ssc install surveybias
Equivalently, you may install from this website:
net from https://www.kai-arzheimer.com/stata/ net install surveybias
How to Use the surveybias Software
surveybias and the underlying methodology are applicable to any survey which samples a multinomial variable whose distribution in the population is known. The package consists of three separate ados. The main command is surveybias. It computes the , , and as well as standard errors and statistical tests from a variable held in memory and additional information about the true distribution of the respective variable in the population.
surveybias is complemented by surveybiasi, an immediate command that makes these calculations based on information typed as arguments on the command line. Using surveybiasi, it is possible to produce estimates of polling accuracy from published margins when the raw data are not available.
surveybiasseries takes this idea one step further. In the aftermath of an election, researchers will often want to compare polling accuracy across time and firms, but commercial pollsters tend to make their raw data available for secondary analysis only after some cooling-off period, if at all. surveybiasseries calculates accuracy measures from a dataset of published margins, where each row represents the headline findings from a single survey. surveybiasseries stores the accuracy measures as new variables in the dataset so that it is very easy to model polling accuracy as a function of variables such as duration and timing of field work, sample size, or polling company.
[by varlist: surveybias varname if in weight , popvalues(#..#) verbose numerical vce() cluster() svy subpop(var) level(#)
surveybias compares the distribution of a categorical variable varname in the dataset to its true distribution in the population. This true distribution is submitted to the command as a numlist in popvalues. Subsamples can be selected via the if and in qualifiers so that the accuracy of group-specific predictions may be assessed. Standard errors will be based on the size of the reduced sample. When using the survey estimator, subpopulations should be specified with the subpop() option instead.
Click here for a worked example that shows how surveybias can be used to assess bias in a single French election poll.
For a separate worked example that demonstrates the use weights and information on the sampling design with surveybias in assessing bias in educational attainment, click here.
surveybiasi , popvalues(#..#) samplevalues(#..#) n(#) numerical level(#)
surveybiasi is an immediate command that compares the distribution of a categorical variable in a survey to its true distribution in the population. Both distributions need to be specified via their respective options.
Click here for an example that shows how surveybiasi can be used to assess bias using published margins from an US pre-election poll.
surveybiasseries if in popvariables(varlist) samplevariables(varlist) nvar(varname) generate(newvarstub) missasnull popvalues numerical descriptivenames
surveybiaseries estimates accuracy measures from a dataset of survey margins. Each observation represents a single poll. For each survey, the distribution of some categorical variable is given by a series of samplevariables. The distribution can be expressed in terms of absolute frequencies, relative, frequencies, or percentages. Information on the true distribution can be specified either directly on the command line via the popvalues option or as another series of variables, specified through the popvariables option. Either popvalues or popvariables must be given, but not both. Moreover, another variable whose name is passed to the program in the nvar option must hold the respective sample sizes. Using the if and in qualifiers, it is possible to restrict the analysis to a subgroup of surveys.
The command leaves behind a series of new variables, whose names are based on the stub submitted via the generate option. generate is required. In these variables, surveybiaseries stores the complete information that would be generated by the equivalent series of surveybias commands.
Click here for a more complex example that shows how surveybiasseries can be used to quickly analyse bias in a large number of polls. Includes estimation of house effects and house-specific bias against a single party.
If you are worried about, or interested in analysing survey bias with Stata, try our software now. It is free and may be quickly installed or updated by entering
ssc install surveybias
Try it now.
3 thoughts on “Survey Bias Quantified: Surveybias Software”
I have a question which it not entirely relevant to this post – I would like to know the statistical model that might validate or refute the accuracy of the Electoral College, versus that popular vote. Actually, there is no need to consider that popular vote: no scientist or mathematician would ever consider accurate the counting of the voters who happened to show up at the polls. The only way anyone could seriously even call it “popular vote” would be if ALL the eligible voters voted, or at least some mathematically defensible percentage – not 55%! However, I am relatively certain that the Electoral College, or something like it, COULD be considered accurate, if the correct predictive statistics were applied. Have you worked on this?
I’m not sure if I understand your question correctly. One could probably model the votes in the EC as draws from a binomial or even multinomial distribution of preferences amongst electors. But in what sense could we assess the accuracy of this vote? The members of the the EC are not a random sample, they are the universe of EC members, at least for any given presidential election year. If we are talking about the true preferences of the electorate, things look even worse than with the popular vote. In this case, the result in each state would constitute a very large but systematically biased (voters vs non-voters) sample, with clustering at the state level. Things are then compounded by some weighing (the number of EC votes is not exactly proportional to the size of the electorates), and by retaining only the modal category within each state (winner takes it all). In short, it is not obvious how the EC, that is loosely based on the popular vote would make the outcome more similar to the unknown will of the electorate than the popular vote itself.