How Do You Know that Poll Is Accurate?: A Look at How Polls Are Conducted
Category: News & Politics
Via: broliver-thesquirrel-stagnasty • 12 years ago • 20 commentsNews Flash! The latest results are in and they show that president Obama has fallen three percentage points in the past two weeks, leaving him at an all time low approval rating.
*******
Several questions spring to mind when you see poll results that make a proclamation of how the residents of the country feel. Amongst them:
-
How is it possible that a small sample size could reflect the attitudes and opinions of hundreds of millions?
-
How do they know that the polling is accurate?
-
How come no one asked me?
I know that some of the community doubt that anything substantial can be established or inferred from the results of polls, so I wanted to look at how polls are taken, how the results are displayed and interpreted, and what, if anything, can be said about those results.
The first thing to do would be to define what a poll is . At its most basic level, a poll is a sampling of a population to determine the occurrence or prevalence of an attitude or attitudes. As attitudes are subject to change, a poll is therefore a look at how that population is feeling at the time the poll is taken. There are several different types of polls, but the most basic form is an opinion poll, and if conducted properly, these can give us a view into what people are thinking at any given moment.
So, knowing what a poll is, we can now shift to the question how are these polls conducted? As it turns out, polling has several rules that, when followed, have been shown to yield very consistent and verifiable results. The first and foremost of these rules is that the sample must be random. That would put the innumerable polls on internet sidebars and the like as inaccurate because they are self selecting . [ Also here ] That is, one sees the poll and is more likely to respond if ones opinion of the polling subject is strong while others may have a less strong position and may therefore not respond, which skews or biases the results.
If the sample must be random, how do the polling agencies achieve this randomness ? Well, as dangerous as it is to make them, some statistically based assumptions about the population being polled need to be made in order to achieve random samples. From the paper How Are Polls Conducted put out by the Gallup organization:
To conduct a national opinion poll, Gallup first chooses a method by which all or most Americans have an equal or known likely chance to be selected. Of course, most Americans have a place of residence, making it immediately obvious that if it were possible to sample for a list of all residential addresses in the country, the principle of comprehensive sampling frame could be achieved.
At first, the Gallup organization (and others as well) sent out people to knock on doors. Over time this became increasingly expensive,so when phones were in most of the households, they switched to a method called Random Digit Dialing, or RDD. This method dials numbers generated by a computer based on active phone exchanges. Since the use of cellphones has become more ubiquitous, they have also been added to the mix. Once somebody answers the phone, in order to make the selection process more random, the person in the household who last had a birthday is asked for rather than the person who tends to answer the phone the most.
When you see the words Margin of Error on a poll, what they are really referring to is the Margin of Sampling Error . This term, frequently abbreviated MOSE, arises from the study of statistics and is, in effect, the price we pay for polling. In short, what it means is the sample size is related to the accuracy, or (from the same article linked last):
The MOSE describes the range that the answer likely falls between if we had talked to everyone instead of just a sample.
For example, for a telephone sample of 1000 randomly selected adults nationwide, the finding from the poll will be within plus or minus 3 percentage points of the answer we would have gotten had we talked to all 210 million adults. So if one of the findings of the poll was that 58% approved of the job their Governor was doing, we would know that the true value would lie somewhere between 55% and 61% if we had talk to the whole population in the state.
To be technically correct, we really only have some degree of confidence around the MOSE we calculate for probability based samples. Generally, pollsters calculate the MOSE using a 95%confidence level. That is, in 95 times out of a 100, we expect the answer we get from the survey is reflective of the true answer within the MOSE.
(There is a wonderful graph and accompanying explanation on the linked page.)
So far we have our random sample and know how many people we will need to talk to in order to achieve a result with a known MOSE and confidence level, now we have to figure out what to ask . Luckily, many pollsters have conducted many polls so we can gain from their experience. In the political arena, it is customary to ask the horse-race question first . This is to ensure that the answer to the most important (to some people, anyway) question is not affected by the preceding questions.
Finally, when we have all of that data, it might still not represent the population that we are aiming for. Since we went through all of this trouble to achieve a nice random sample, how come we have to go further? Well, if you are talking about a political horse-race poll, the group of people whom you want to be polling is the people who will turn out to vote. This is where all that metadata about the respondents such as your age, income, degree of education, etc. come into play. After the random sample is achieved and recorded the polling data is weighted. You can see why here and here .
Pollsters go through all of these steps to assure that their numbers are close to reality because they will be judged on the correctness of their polls. Reputable polling agencies live or die by the accuracy of the numbers that they produce. Some polls, however, are designed as propaganda tools and are not intended to represent all of a population. These polls you will often see on the sidebars of your window, flashing at you to Vote Now or some such.
How come no one asked you? You probably slammed down the phone shouting, Dammit, those people can never leave me alone! thinking it was just another intrusive political phone call.
_____________________________________
Here are some good resources for further reading:
-
SOURCES OF VARIATION IN PRE-ELECTION POLLS: A PRIMER Or Why Different Election Polls Sometimes Have Different Results : Cliff Zukin, Professor of Public Policy and Political Science, Bloustein School of Planning and Policy & Eagleton Institute of Politics, Rutgers University
-
How Stuff Works.com: Political Polling
-
Rasmussen Reports FAQ page
-
Gallup FAQ Page
-
American Association of Public Opinion Research resource page.

I know from some ofthe conversations that I have read on here that there are some people who do not believe that a poll can be accurate. I hope that this article can make them see how polls can indeed be accurate, at least relatively so.
Good article . It covers quite a lot of the details involved in proper poll taking .
Thank you, Petey.
Just barely brushed the surface, really. There are a lot of resources devoted to polling, and not just the political type. Some people tend to be dismissive of polls when they really should not be because there is quite a bit of established science and statistical methodology involved. I think part of the reason that they are is the prevalence of all these trash polls that pop up on the sidebars of their computer screens. Those polls have serious methodological flaws and are not really representative of the population at large.
Poll taking is as much an art as it is a science . Methodological flaws can involve the venue [as you pointed out ] the form of the questions , their sequence [as you pointed out ] and the manner of the poll taker , among other things . Its a tricky business .
What polling companies are considered reliable, accurate, honest? Gallup? Pew?
Reliability is usually indicated by good agreement with other similar polls .
Many polls are basically like weather vanes and have a short life. It shows the current direction of the wind. As the wind direction changes (so to speak), so to does the popular opinion.
Polls can also be milestones whereby you can compare opinions (and facts) over a long range of time.
Polls also can be very wrong. A good example would be the Gallup poll which showed Romney leading Obama, 49 to 48 percent, in the 2012 elections.
In all cases; Caveat emptor.
So this is how polls work! Good to know, Brolly. Maybe I won't slam down the phone the next time I see a caller on my caller ID that looks like a polling organization! I have actually answered a few of them over the past couple of years.
Broliver
Excellent article.
I have a simple (some would say cavalier) attitude towards polling of any kind and political polling in particular.
In my view, the poll results should be looked at like the results of asingle blood pressure test. The numbers provided will not definitively tell you if you have heart disease (or an impending heart attack) but will give you information that you should use in formulating in your own mind a decision based on those numbers, historical insight that you possess and input from those you respect and trust on whether or not you should see a physician or not.
Simplistic I know, but the bottom line poll results are data and individuals must interpret that data rather than simply accept what general consensus is as defined by the experts.
Again thanks for a very informative article
Gallup polls have a standard margin of sampling error of 4% at a 95% confidence level, so that would mean that Candidate Romney could have had anywhere from 46-53% and President Obama could have been between 45-52%. The final results of the election were 51.1% for the president and 47.2% for the challenger.
So, the poll wasn't wrong, per se, it was just beyond the resolution of the poll to show who the winner would be.
A poll is just a tool for taking the pulse (to continue the analogy) of whatever population is being polled.
Any poll results should be viewed with a very large grain of salt, because they are better indicators of trends than absolute numbers. But don't let that large grain of salt affect your cardio-vascular health....(It's ok. You can groan at that one!)
Slyly humorous but also correct
Thanks
Here are some things to watch out for regarding Polls and Research :
Taken From The page Linked above and Here .
All of these practices are frowned upon, to put it lightly, by the various associations of firms that conduct polling because of the dilution of the public's trust in polls.
Exactly Broliver . If one used all the numbers one could compute the probabilty of a win for each candidate . Since the results were so close in the poll the probability of winning was going to be close as well .
Polls reflect opinions, but how much do opinions really matter? Hmmm, that is the question I often ponder.
Politicians would care if they seek re-election. Those who represent their constituencies would want to act in the manner opined by the majority of voters. For example, annual Gallup polls indicate that a vast majority of voters favour Israel exponentially over the Palestinians which invariably will lead to their government representatives supporting Israel if they have any desire of being re-elected.
Polls can be utilized for many things, not just political campaigns and gathering opinions about political issues.
If you live in a city of 50,000 and you want to know the market penetration of the widget that you make, or are doing market research to see if a certain business move is viable, I would imagine a poll could be designed to show you that information.
If you are thinking more deeply, then there is no right answer to the question, Do Opinions Matter?... That would vary by the individual, I think.
You obviously did not read the article I had linked to. Gallup's failure had nothing to do with their already flawed margin of error.
Grrrrrrrrrrrrroaaaaaaaaaaaan! I need to measure my pulse rate about now...and I didn't even have a large grain of salt
.
I stick by my response above until such a time as it is shown to be incorrect. Did you read my article and the associated links? It would appear that you did not.
I did read the article. The results are standard. Perhaps you could point out where you saw this damning evidence of something pernicious? Also, could you please explain how the Margin of Sampling Error that Gallup uses is flawed?