Margin of error

This page discusses the use of the term in opinion polling. For other uses, see margin of error (disambiguation).

The margin of error is an estimation of the extent to which a poll's reported percentages would vary if the same poll were taken multiple times. The larger the margin of error, the less confidence one has that the poll's reported percentages are close to the "true" percentages, i.e. the percentages in the whole population.

The margin of error can be calculated directly from the sample size (the number of poll respondents) and is commonly reported at one of three different levels of confidence. The 99 percent level is the most conservative, the 95 percent level is the most widespread, and the 90 percent level is rarely used. Formally, if the level of confidence is 99 percent, one is 99 percent certain that the "true" percentage in a population is within a margin of error of a poll's reported percentage for a reported percentage of 50 percent. Equivalently, the margin of error is the radius of the 99 percent confidence interval for a reported percentage of 50 percent.

Note that the margin of error only takes into account sampling error. It does not take into account other potential sources of error such as bias in the questions, bias due to excluding groups who could not be contacted, people refusing to respond or lying (selection bias), or miscounts and miscalculations.

Calculations and caveats

The margin of error is a simple re-expression of the sample size, N. The numerators of these equations are rounded to two decimal places.

Margin of error at 99 percent confidence <math>\approx 1.29/\sqrt{N}\,<math>

Margin of error at 95 percent confidence <math>\approx 0.98/\sqrt{N}\,<math>

Margin of error at 90 percent confidence <math>\approx 0.82/\sqrt{N}\,<math>

The margin of error is not fully defined if the confidence level is not reported. If an article about a poll does not report the confidence level, the margin of error can be calculated to a desired degree of confidence given the reported sample size. Also, if the 95% margin of error is given, one can find the 99% margin of error by increasing it by about 30%. If an article reports neither the confidence level nor the sample size, readers should only assume a particular level of confidence for casual, low-stakes interpretations.

The margin of error is a poll-level statistic that should not be used to evaluate or compare reported percentages. However, due to its unfortunate name (it neither establishes a "margin" nor is the whole of "error"), it has become one of the most widely overinterpreted statistics in general use by the media. It is frequently misused to judge whether one percentage is "significantly" higher than another or to specify the error associated with reported percentages outside of 50 percent.

Understanding the margin of error

A running example

This running example from the 2004 U.S. presidential campaign will be used to illustrate concepts throughout this article. It should be clear that the choice of poll and who is leading is irrelevant to the presentation of the concepts. According to an October 2 survey by Newsweek, 47 percent of registered voters would vote for John Kerry/John Edwards if the election were held today. Forty-five percent would vote for George W. Bush/Dick Cheney, and 2 percent would vote for Ralph Nader/Peter Camejo. The size of the sample is 1,013, and the reported margin of error is ±4 percent. The 99 percent level of confidence will be used for the remainder of this article.

The basic concept

Polls require taking samples from populations. In the case of the Newsweek poll, the population of interest is the population of people who will vote. Since it is impractical to poll everyone who will vote, pollsters take smaller samples that are intended to be representative, that is, a random sample of the population. It is possible that pollsters happen to sample 1,013 voters who happen to vote for Bush when in fact the population is split, but this is very, very unlikely given that the sample is representative.

Given the size of the sample (1,013), probability theory allows the calculation of the probability that the poll reports 47 percent for Kerry but is in fact 50 percent, or is in fact 42 percent, or is in fact zero percent. This theory and some Bayesian assumptions suggest that the "true" percentage will probably be very close to 47 percent. The more people that are sampled, the more confident pollsters can be that the "true" percentage is closer and closer to the observed percentage. The margin of error is a rough, poll-wide expression of that confidence.

Statistical terms and calculations

The margin of error is just a specific 99 percent confidence interval, which is in turn a simple manipulation of the standard error of measurement. This section will briefly discuss the standard error of a percentage, briefly discuss the confidence interval, and connect these two concepts to the margin of error.

The standard error can be estimated simply given a proportion or percentage, p, and the number of polled respondents, N. In the case of the Newsweek poll, Kerry's percentage, p = 0.47 and N = 1,013. Given some statistical theory outlined below, the following holds:

Standard error = <math>\sqrt{\frac{p(1-p)}{N}}<math>

The standard error (.016 or 1.6 percent) helps to give a sense of the accuracy of Kerry's estimated percentage (47 percent). A helpful, Bayesian interpretation of the standard error is that the "true" percentage (unknown) is located somewhere around the estimated percentage (47 percent). The standard error can be used to create a confidence interval within which the "true" percentage should be to a certain level of confidence.

Plus or minus 1 standard error is a 68 percent confidence interval, plus or minus 2 standard errors is approximately a 95 percent confidence interval, and a 99 percent confidence interval is 2.58 standard errors on either side of the estimate.

The margin of error is the radius (half) of the 99 percent confidence interval, or 2.58 standard errors, when p = 50 percent. As such, it can be calculated directly from the number of poll respondents.

Margin of error (99%) = 2.58 × <math>\sqrt{\frac{0.5(1-0.5)}{N}} = \frac{1.29}{\sqrt{N}}<math>

To conclude, the margin of error is the 99 percent confidence interval for a reported percentage of 50 percent. If p moves away from 50 percent, the confidence interval around p will be smaller. Thus, the margin of error represents an upper bound to the uncertainty; one is at least 99 percent certain that the "true" percentage is within a margin of error of a reported percentage for any reported percentage.

The use and abuse of the margin of error

The margin of error grew out of a well-intentioned need to compare the accuracy of different polls. However, its widespread use in high-stakes polling has degraded from comparing polls to comparing reported percentages, a use that is not supported by theory. A web search of news articles using the terms "statistical tie" or "statistical dead heat" returns many articles that use these terms to describe reported percentages that differ by less than a margin of error. These terms are misleading; if one observed percentage is greater than another, the true percentages in the entire population are probably ordered in the same way. In addition, the margin of error as generally calculated is applicable to an *individual percentage* and not the difference between percentages (the margin of error applicable directly to the "lead" is approximately equal to twice the generally stated margin of error - this is exactly the case only for a two-choice poll with a result of 50% for each choice). The margin of error is often interpreted as if the poll gives either no information (a difference within a margin of error) or perfect information (a difference larger than a margin of error) about the ranking of two percentages in the population. As the margin of error continues to be inappropriately applied, simpler alternatives (sample size) or more complex alternatives (standard error, probability of leading) may be warranted.

Incorrect interpretations of the margin of error

Here are some INCORRECT interpretations of the margin of error based on the Newsweek poll.

• Kerry and Bush are "statistically tied" or are in a "statistical dead heat".
  • It only "matters" if Kerry leads Bush (or vice versa) by more than 4 percent.
• Any change in the percentages for future polls does not "matter" unless it is more than 4 percent.
• Because Nader got 2 percent and the margin of error is 4 percent, he could potentially have 0 percent.
• The margin of error is the same for every percentage, i.e. 47% ± 4%, 45% ± 4%, 2% ± 4%.

Arguments for the use of the margin of error

• For casual comparisons of different polls, it is helpful to define a benchmark (99 percent confidence interval radius for an estimated percentage of 50 percent) that reflects the sampling variance of the poll.
  • Sampling variance does not decrease linearly with increasing numbers of respondents (it decreases by the square root of N), so using the number of respondents as an inverse measure of standard error might be confusing.

Arguments against the use of the margin of error

• The margin of error is a simple transformation of the number of respondents into an ambiguous term that is neither a "margin" nor the whole of "error".
• The margin of error is being confused with the confidence interval of reported percentages.
  • The 99 percent confidence interval radius is smaller than the margin of error for any percentage besides 50 percent.
  • It is much smaller and more asymmetric for very high and very low percentages.
• It is not a "margin" at all; the probability of the true percentage being outside the margin of error is low but nonzero.
• There is no agreed-upon confidence level. Most pollsters use 99 percent, but many use 95 percent or 90 percent; this makes their polls look more accurate.
• When the purpose of polls is to compare percentages, the use of the margin of error is tempting but inappropriate.
• Perhaps most importantly, there are many different sources of error in polling, and variance due to sample size is not likely to be the only contribution. Other possible contributions to error include:
  • Sampling bias, when the sample is not a representative sample from the population of interest.
  • The phrasing of the question may not be appropriate for the conclusions of the poll.
  • Response error (Sudman & Bradburn, 1982)
    • Deliberate distortion (fear of consequences, social desirability, response acquiescence).
    • Misconstrual (not understanding the question).
    • Lack of knowledge (guessing to try to be helpful).

Margin of error and population size

An interesting mathematical fact is that the margin of error depends only on the sample size and not on the population size, provided that the population is significantly larger than the sample size. Thus for instance, the poll in the running example with 1,013 randomly sampled registered voters would yield essentially the same margin of error (4% with a 99% level of confidence) regardless of whether the population of registered voters consisted of 100,000 people or 100,000,000 people.

This may seem counter-intuitive at first; after all, each person in the population has a unique personality and opinion, and in a very large population, only a very small fraction of such people would actually be polled, and it would thus seem that the poll is not capturing enough information. However, because a poll involves only a very specific question, there is only one relevant attribute in the population that needs to be considered, and this means that an individual's opinion is effectively equivalent to those of many other members of the population, some fraction of which will be polled. For instance, in the running example, the only relevant attribute of a population member is whether he or she is a Bush voter, a Kerry voter, or a Nader voter - all other characteristics of a population member are irrelevant. Thus for instance if there are 100,000,000 registered voters, and 48,000,000 of them were Kerry voters, then for the purposes of this statistical analysis all of the 48,000,000 individuals in this group would be completely interchangeable and equivalent. An individual Kerry voter may fret that because he or she was not polled directly, his or her opinion was not reflected in the results of the poll; but this voter has 47,999,999 other voters with identical opinions (as far as the poll question is concerned), and it is exceedingly likely that a poll of 1,013 voters will contain a properly representative fraction of this group, provided of course that the voters being polled were selected randomly.

To give an analogy, suppose that one is trying to estimate the percentage of salt in an ocean. This can be easily accomplished by taking a glass of seawater and then chemically analyzing the proportion of salt in that sample. The amount of salt and water in this glass is far smaller than the amount of salt and water in the ocean under study. Nevertheless, the sample is likely to give a very accurate measurement of the ocean's salinity, provided of course that the salt is evenly distributed across the ocean (this hypothesis is the analogue of the hypothesis that the poll sample is being randomly chosen). In fact, one could already obtain a crude but reasonable estimate of salinity by testing just a single drop of seawater, though of course the larger sample in the glass would provide a more accurate measurement. This analogy may help explain why it is the sample size, rather than the population size, that determines the margin of error in a poll.

Comparing percentages: the probability of leading

Tables

The margin of error is frequently misused to determine whether one percentage is higher than another. The statistic that should be used is simply the probability that one percentage is higher than another. This can tentatively be called the Probability of Leading. Here is a table that gives the percentage probability of leading for two candidates, in the absence of any other candidates, assuming 95% confidence levels are used:

For example, the probability that Kerry is leading Bush given the data from the Newsweek poll (a 2% difference and a 4% margin of error) is about 68.8%, provided they used a 95% confidence level. Note that the 100% entries in the table are actually slightly less. Here is the same table for the 99% confidence level:

If the Newsweek poll used a 99% confidence level, the probability that Kerry is leading Bush rises to about 74.1%. It is evident that the confidence level has a significant impact on the probability of leading.

Derivation

The rest of this section shows how the Newsweek percentage might be calculated. This probability can be derived with 1) the standard error calculation introduced earlier, 2) the formula for the variance of the difference of two random variables, and 3) an assumption that if anyone does not choose Kerry they will choose Bush, and vice versa, i.e. they are perfectly negatively correlated. This assumption may not be tenable given that a voter could be undecided or vote for Nader, but the results will still be illustrative.

The standard error of the difference of percentages p for Kerry and q for Bush, assuming that they are perfectly negatively correlated, follows:

Standard error of difference = <math>\sqrt{\frac{p(1-p)+q(1-q)+2pq}{N}}.<math>

Given the actual percentage difference p − q (2 percent or .02) and the standard error of the difference, above (=.03), use a program like Microsoft Excel to calculate the probability that .02 is greater than zero given a normal distribution with mean 0 and standard deviation .03.

These calculations suggest that the probability that Kerry is "truly" leading is 74 percent.

More advanced calculations behind the margin of error

Let N be the number of voters in the sample. Suppose them to have been drawn randomly and independently from the whole population of voters. This is perhaps optimistic, but if care is taken it can be at least approximated in reality. Let p be the proportion of voters in the whole population who will vote "yes". Then the number X of voters in the sample who will vote "yes" is a random variable with a binomial distribution with parameters N and p. If N is large enough, then X is approximately normally distributed with expected value Np and variance Np(1 − p). Therefore

<math>Z=\frac{X-Np}{\sqrt{Np(1-p)}}<math>

is approximately normally distributed with expected value 0 and variance 1. Consulting tabulated percentage points of the normal distribution reveals that P(−2.576 < Z < 2.576) = 0.99, or, in other words, there is a 99 percent chance of this event. So,

<math>P\left(-2.576<\frac{X/N-p}{\sqrt{p(1-p)/N}}<2.576\right)=0.99.<math>

This is equivalent to

<math>P\left(\frac{X}{N}-2.576\sqrt{\frac{p(1-p)}{N}}

\frac{p(1-p)}{N}}\ \right)=0.99.<math>

Replacing p in the first and third members of this inequality by the estimated value X/N seldom results in large errors if N is big enough. This operation yields

<math>P\left(\frac{X}{N}-2.576\sqrt{\frac{(X/N)(1-(X/N))}{N}}

The first and third members of this inequality depend on the observable X/N and not on the unobservable p, and are the endpoints of the confidence interval. In other words, the margin of error is

<math>2.576\sqrt{\frac{(X/N)(1-(X/N))}{N}}.<math>

References

• Sudman, Seymour & Bradburn, Norman (1982). Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey Bass.

External links

Template:Wikibooks

• Newsweek poll, October 2, 2004 (http://msnbc.msn.com/id/6159637/site/newsweek/)
• Google News search for news articles containing "statistical tie" (http://news.google.com/news?hl=en&lr=&tab=wn&ie=UTF-8&scoring=d&q=%22statistical+tie%22+OR+%22statistical+dead+heat%22+OR+%22statistically+tied%22&btnG=Search+News)fr:Marge d'erreur