What makes a poll scientific?
1936 was the most important year in the two centuries-long history of opinion polling. It was the year that George Gallup’s sample poll showed that improved methodology could yield better results than what had come before.
Today, the principles established by Gallup and his pioneering colleagues Elmo Roper and Archibald Crossley are so entrenched that they are described with one of the most powerful words in the English language: scientific.
If something is scientific, the likelihood of its truth is so much the greater. This is why Engels described Marx’s ideas as scientific socialism, L. Ron Hubbard named his church of Scientology, and professors of politics everywhere call themselves political scientists. The word itself gives authority.
So what is it that makes an opinion poll scientific?
The scientific poll is first and foremost described as a sample poll. The laws of statistics show that from a relatively small sample it is possible to extrapolate the characteristics of a much larger population. To conduct a census or an election is an exercise that can cost billions. A sample poll gives the promise that a similar result can be achieved with a much lower expense of time and money.
However, the term sample poll is itself problematic. After all, every poll is conducted amongst a “sample.” The many millions who participated in the famous Literary Digest fiasco, the hundreds of thousands who have given their opinion on the Yahoo! homepage, and the thousands of Ron Paul supporters who can be counted on to flood online ballot boxes with votes for their hero. All of these groups constitute samples, though their validity is certainly in question.
A key idea in scientific polling is randomness. A pollster attempts to extract information from a “random” sample. She does not knock on a thousand doors in the same apartment complex, or dial a thousand consecutive phone numbers, or take responses from a thousand volunteers. Instead, using methods to randomize her search for respondents, trying desperately to limit the grave problems of self-selection and non-response, she feels she is able to construct the holy grail of polling, a representative sample.
Representativity is the single most important aspect of a scientific poll. The makeup of the poll should as much as possible reflect the makeup of the population. If African Americans make up 14% of the national population, then they should make up 14% of a national poll’s sample. If the population is 51% female, then so too should be the sample. Pollsters will actually weight the responses given by different demographic groups in order to to bring them into line with the proportions laid out in census data. The results of any poll, no matter how random the sample, are necessarily massaged in order to give the right data.
The quest for representativity illustrates the fatal flaw of scientific polling. It is true, the laws of statistics tell us that if we imagine a vat filled with one million marbles, some red and some blue, and that if we draw out a sample of one thousand, we will be able to tell their relative proportion in the whole, plus or minus three percent, nineteen times out of twenty. But can we be so certain when we turn these laws on the minds and souls of human beings, so much more diverse and complex?
We cannot. The author of this article has a university degree, a Canadian passport, and a decently filled-in beard. Can he be said to speak for all those who share these attributes? Obviously not, yet this is still the daily practice of the scientific pollsters.
In the exact sciences, the conditions of an experiment can be controlled. They can be repeated by other scientists in other labs. The apparatus and methodology are shared publicly and challenged by others. Opinion polling, quite clearly, is not an exact science. An opinion poll is nothing more than a collection of data.
Does opinion polling require skill, art, technique, experience, and expertise? Can it deliver valuable insight? Of course.
Is it science? The answer is no.