BIAS IN SLOVENE OPINION POLLING: MORE ABOUT

UNCLAS LJUBLJANA 000830

SIPDIS


SENSITIVE

DEPT FOR EUR/PPD, EUR/NCE AND INR

E.O. 12958: N/A
TAGS: KPAO PGOV PINR SI
SUBJECT: BIAS IN SLOVENE OPINION POLLING: MORE ABOUT
AFFORDABILITY THAN POLITICS?


Sensitive but unclassified -- protect accordingly


1. (SBU) SUMMARY. As in most countries, the results
of public opinion polls in Slovenia are widely
discussed among media, politicians, pundits and the
public. However, few of these "consumers" are aware of
the methodologies that are used or the ease with which
polling data can be manipulated. Although
misrepresentation of polling data exists in virtually
every country, the close relationships between polling
centers, media outlets, and political parties in
Slovenia make it incumbent on the consumer of opinion
polls to study their methodologies closely and take all
results with a grain of salt. In an analysis of
polling methodologies used by different survey research
centers in Slovenia, we have determined that
professional pollsters are highly knowledgeable and
cognizant of potential biases inherent in their polling
methods. However, lack of sufficient funding in a
small country often means that polls are done on the
cheap and that many of the results contain considerable
bias. More difficult to determine is whether bias is
ignored intentionally and whether political parties can
"commission bias" when paying for a poll. END SUMMARY.


2. (U) In late August, Post's Public Diplomacy Officer
(who has a PhD in Political Science and a Master's-
level certificate in Survey Research Analysis)
conducted a series of interviews with researchers at
three major polling centers in Slovenia: the Research
Center for Public Opinion and Mass Communication at the
University of Ljubljana; the DELO STIK polling agency
affiliated with the leading Slovenian daily "Delo"; and
the NINA MEDIA polling agency affiliated with the daily
"Dnevnik." The following observations and comments are
based on the interviews mentioned above and an analysis
of the methodologies used by these polling centers.

-------------- --------------
Sample Bias vs. Sampling Error: A crucial distinction
-------------- --------------


3. (U) One of the main problems with public opinion
polls in Slovenia and one of the main causes of faulty
analysis by the media lies with the fact that polling
agencies often do not take adequate care to ensure that
various biases are eliminated from their polling
methodologies. Every opinion poll contains some bias
and reducing bias to a minimum often comes at a great
cost. Thus, every polling agency faces a tradeoff
between bias and the cost of utilizing expensive
procedures to minimize it. Since most polling agencies
are businesses looking to make a profit, cost-cutting
is prevalent; nevertheless, lower costs often mean less
accuracy.


4. (U) Another major problem is that the media, who
are the primary consumers and interpreters of opinion
polls, often do not adequately understand the
distinction between sample bias and sampling error.
Opinion polls that have considerable bias but that have
minimal error are often presented to the public as if
they were quite accurate. Typically, for example, a
media outlet will report that an opinion poll has a
margin of error of plus or minus 2-5 percent. What
they do not say, however, is that in addition to this
margin of error (known as "sampling error"),these
polls also--in fact invariably--contain sample bias,
which can often skew a given poll by as much as 10, 15,
or even 25 percent. A recent conversation with the
news director of a major Slovenian television station
revealed a complete failure to appreciate this crucial
distinction.


5. (U) Sampling error is a measure of the likelihood
that a perfectly random sample of the population will
have the same characteristics as the population as a
whole. Sampling error can be measured using a
statistical formula. Sample bias, however, cannot be
measured. Sample bias occurs when a polling agency
consistently fails to obtain perfectly random samples
of the population (often for a multitude of different
reasons). For example, if a polling agency were to
interview people only within a 50-mile radius of their
headquarters, they would consistently fail to obtain a
perfectly random sample of the population. However,
extensive polling within this 50-mile radius could very


well reduce the margin of error to plus or minus 2
percent, or even less.


6. (U) Sample bias is almost impossible to eliminate.
Seemingly innocuous methodologies like phoning random
numbers to conduct surveys can produce remarkably
biased results. For example, poorer households tend
not to own telephones (or subscribe to fewer lines),so
phone interviews consistently bias the sample in favor
of wealthier respondents, who in turn often have
different voting preferences. On the other hand, if
telephone polls are conducted during the day, they may
consistently exclude working professionals (since they
are less likely to answer their home phone) and thus
overstate the views of the elderly, students, and the
unemployed. While telephone penetration rates in
Slovenia are estimated at 94 percent, and while most of
the centers poll from 3:00-9:00 PM, the prevalence of
telephone bias still exists and impacts polling
results. This is true especially since "busier"
members of household are less likely to answer the
phone while those who have more free time are likelier
to agree to participate in a survey.


7. (SBU) Polling agency directors were of course
cognizant of such methodological flaws, but noted that
telephone polls were cheaper than fieldwork and
indicated that the exorbitant costs of fieldwork often
prevented them from doing more methodologically
rigorous polling. Polling on the cheap, they argued,
was better than no polling at all. While it is true
that some media in the U.S. regularly conduct
methodologically substandard polls, these polls are
usually discounted by polling professionals, who rely
on the far more accurate analyses provided by academic
institutions, major media outlets, and reputed polling
organizations like Gallup, who have the financial means
to conduct methodologically rigorous surveys. In
Slovenia, the small number of polling centers and the
modest funding they receive means that few polls employ
a rigorous methodology.

--------------
Weighted Samples and Representative Regions
--------------


8. (U) One standard method for reducing bias involves
the "weighting" of survey samples. This involves
factoring into the analysis the known differences
between the sample and the population as a whole
(usually determined from a census). For example, if
you randomly telephone ten people and the result is
that 3 women and 7 men answer the phone, and yet you
know the population is split evenly between men and
women, then to weight the sample you would simply
multiply the responses of the women by 7/10 and the
responses of the men by 3/10. The problem with this
method, however, is that while it may be easy to weight
responses by gender or area of residence, there are
hundreds of other variables that simply cannot be
weighted because their underlying values are unknown
(e.g. religiosity, previous affiliation with the
Communist Party, etc). Most Slovenian research centers
weight their samples by gender, education, and region
of residence. However, other variables like affluence,
church attendance, and previous support for the
Communist regime have all been shown to have a
significant--and perhaps even greater--impact on
political preferences than the variables currently in
use.


9. (U) Another method for minimizing bias is to pick a
representative region based on prior election results.
This is sometimes called the "Peoria method." If a
particular region or municipality within a country had
election results that perfectly mirrored the national
averages, then polling agencies will sometimes conduct
methodologically rigorous polls within those regions in
order to predict future election results. The problem
with this method, however, is that populations shift
and what may be a representative region during one
election cycle may no longer be representative during a
subsequent cycle. This is especially true if a new set
of issues is being raised from one election to the
next.

--------------


Slovenia's EP elections: What Went Wrong
--------------


10. (SBU) The failure of most Slovenian polling
agencies to adequately predict the European
Parliamentary elections is mainly due to various
different types of sample bias. While it is impossible
to authoritatively determine why such bias occurred,
one hypothesis that seems likely is that the winner of
the EP elections--Nova Slovenija (NSi or New Slovenia)-
-has a rather "unrepresentative" electorate. To cite
but one example, NSi attracts both less educated voters
and highly educated voters. So, if a polling agency
samples a population and weights the sample by
education, they may actually be missing the two
extremes--the uneducated and the highly educated--and
hence the result may under-represent the percentage of
voters who would vote for NSi.


11. (SBU) Anecdotal evidence from field polls
conducted by DELO STIK also indicates that New
Slovenia's voters tend to be more fearful of the
government (a relic of the Communist system),and hence
less likely to answer public opinion polls out of fear
that the government may be trying to keep tabs on them.
If such suspicions really do exist, they could
potentially result in a large understatement of the
number of NSi voters within a given region. This sort
of bias is also more likely in a telephone poll than in
a field survey.


12. (SBU) The inability of Slovenian polling agencies
to adequately predict the turnout for the EP elections
also reflects a different type of bias, one associated
with the way the survey questions are phrased and
perceived by respondents. It is conventional wisdom
that when asking respondents if they plan to vote in an
election, 10-15 percent more respondents will claim
they intend to vote than actually do. Such "social
desirability bias" (as it is commonly known) stems from
their desire to appear as conscientious citizens. A
poll conducted by the Center for Public Opinion and
Mass Communication indicated that even after the
election was over, 12 percent more respondents
indicated that they had voted than actually did. While
polling center researchers were well aware of this bias
and generally predicted the low turnout in the EP
elections rather accurately, the media seemed more
surprised that so many fewer voters would show up than
had claimed they would vote.


13. (SBU) A final reason for the poor polling results
in the run-up to the EP elections also has to do with
the problem of low voter turnout. Many of the polling
center directors we spoke with claimed that NSi voters
were underrepresented in the pre-election voting
because they are much more disciplined than the
electorates of other parties (meaning that they are
more likely to show up on election day--rain or shine).
Since this is a quality that is difficult if not
impossible to measure prior to election day itself, the
pre-election polls generally overrepresented the other
electorates and underrepresented NSi.

--------------
The Tail Wagging the Dog?
--------------


14. (SBU) The bias inherent within different polling
methodologies and its effect on the popularity of
political parties is not unknown to polling agencies.
Researchers at the Center for Public Opinion and Mass
Communication told us that without properly weighting a
sample for education, the results consistently favor
right-of-center parties. From these and other
comments, we believe that the effects of different
methodologies are well known to those conducting survey
data analysis in Slovenia.


15. (SBU) Gossip and conspiracy theories abound in
Slovene political circles. Polling results are not
immune. The chatter that we hear from our contacts,
particularly among opposition members, is that polling
is notoriously unreliable. The result, they say, is
that the ruling coalition gets a boost from those who
want to go with the winner. According to one source,
the ruling LDS party commissions their own "real" polls


for internal use that are cannily accurate, predicting
the actual July European Parliament election results of
one party to within half a percentage point.


16. (SBU) The director of Nina Media also admitted
that he was personally friendly with many members of
LDS and that they frequently joked with him about the
results of his polls. Although he categorically denied
that pressure was ever put on him to fudge the results,
he admitted that most political parties did maintain
regular contact with his polling agency and that they
sometimes commissioned their own polls. He also
confirmed that most parties are keen to benefit from
higher polling results. As he said, "no one wants to
look unpopular."

--------------
Comment
--------------


17. (SBU) The level of knowledge and expertise among
professional pollsters in Slovenia is very high.
However, public opinion polls are extremely easy to
manipulate by expert pollsters. Polling agencies that
compete for contracts have an incentive to maintain a
reputation for accurate polling, but that does not mean
that other considerations do not come into play. In
Slovenia, polling agencies are generally eager to cut
costs and conduct polls on the cheap. Since the more
accurate methodologies are usually the most expensive--
requiring field interviews and persistence in tracking
down randomly chosen respondents--it is natural for
agencies to sacrifice accuracy in order to cut costs.
However, it is also true that if one wanted to bias a
sample in favor of a particular political constellation-
-either left-of-center or right-of-center--it would be
very easy to do so. In fact, intentional bias of this
sort could easily be attributed simply to the high cost
of a more rigorous methodology.


18. (SBU) While election results generally validate
the accuracy of a given polling agency, it is not
impossible for inflated or deflated pre-election polls
to encourage or dissuade potential voters and thus
influence the result itself. Generally, therefore, one
needs to take all polling results, and claims of
manipulation, with a grain of salt and be aware of the
fact that it is always easy to explain inaccuracies
after an election is over. Parliamentary elections in
Slovenia will be held October 3.

HAAS


NNNN