Yesterday we had a municipal election here in Vancouver. Today the Vancouver Sun published a special Sunday edition to report on the results. The top story in section B, “Chinese voters backed Sullivan” by Miro Cernetig, drew some highly dubious conclusions from tiny sample groups. I know I’ve complained about polls before, but I couldn’t stop myself from writing a letter to the editor. It’s after the jump. [more]
Dear Mr. Cernetig:
The Vancouver Sun completed an exit poll of 375 voters. 14% percent, or about 52 people, identified themselves as Chinese. Of those 52 people, 71% or about 37 people, report voting for Sam Sullivan.
Based on the word of 37 people, you’ve concluded that Sam Sullivan “received massive support from the city’s voters who are of Chinese heritage”. You’ve boldly extrapolated 37 to “roughly 14,000 voters”. Any statistician will tell you that you’re playing hard and fast with the numbers.
Did you, for example, report that the margin of error of a sample group of 52 people out of 19,000 Chinese voters is an astounding 14%? That means the variance of Chinese support for Sullivan could have been as high as 85% or as low as 57%. Isn’t it highly irresponsible to draw conclusions from such a tiny data set, particularly when you fail to report on its size or margin of error?
As it does for each election, the editors at the Sun distort statistics to fit the story they want to tell. When I emailed Managing Editor Kirk LaPointe with similar concerns during the federal election, he said “we’ve worked to de-emphasize poll results in our election coverage.” That’s not evident in the 8 full-colour pie-charts on page B1. When will you and your colleagues stop intentionally deceiving your readership?
Sincerely,
Darren Barefoot
DarrenBarefoot.com 
Go Darren!
Sun researchers are lazy
good job darren, telling them to get their facts straight
Mandatory nitpickery: you’re referring to the 95% confidence interval being roughly 14% for that sample size.
There are other problems with exit polling. First and most obviously, they’re harder to conduct, being as media activities in and near polling places on election day are regulated. The second is to get a representative sample from across the city to cover all sides of the northwest-southeast political axis.
It’s useful to get a sense of how much specific communities or demographic characteristics support the various candidates. I’m not convinced that exit polls will tell the story unless many media outlets pooled their money to pay for a study that hit all corners of the city, and surveyed enough voters to get subsamples that were large enough to be useful. That’s especially important in an election thiat was as close as the one we had.
Even among the “ethnic Chinese” I suspect there’s a gap between recent immigrants and those who are third-generation Canadians. A study that finely grained is beyond the resounces of most newspapers; it tends to be the domain of academics or political pros.
My horse sense tells me that Sullivan did outpoll Green amongst Chinese-Canadian voters due to his knowledge of Cantonese and the fact that recent Chinese immigrants skew to the right, but whether it’s 71% or not I don’t know. Certinaly not enouigh evidence there to call Sullivan’s advantage ‘massive.’
Seems to me that Sullivan’s advantage was that someone with an office in the NPA building had the name James Green.
Ian: I’m no mathematician–can you explain a little more about the nitpickery? I’m not being argumentative–I want to understand what you mean.
The NPA only bought media time in the Cantonese and Mandarin media outlets. In other words they concentrated their outreach to a part of the community that breaks their way and ignored the rest. Certainly COPE and Vision didn’t run perfect campaigns as there wouldn’t have been two groups with inherent compromises in that more ideal case. But someone should really look at just where James Green campaigned (i.e. was he trying to play the spoiler for calculated effect) as his effort just looks more malign the more I read about it.
No prob — I’m no mathematician myself, but I did sail through stats well enough, found it interesting, and still use ’em occasionally.
What people are usually referring to is the sampling error that one has to live with; this is ritualistically expressed as being within x% of true, x% of the time. The “margin of error” referred to in most poll data means that the probability is 95% that the results will be within however-much-percent of the population as a whole; this figure is derived from figuring out how wide of an interval one must establish to cover 95% of the sampling distribution; as it turns out, that’s 1.96 standard deviations in a normal distribution, which leads to the handy-dandy equation of 0.98/sqrt(N) where N is the sample size to figure out the magnitude of the 95% confidence interval, or what’s popularly called the margin of error. Plug 1000 into the equation, and you get 0.031, which is why a poll of 1000 voters is usually stated as having a margin of error of +/- 3.1%. Seriously — that’s all there is to it.
The other 5% of the time, the sample results will fall outside this interval and be even less representative. (In that case, the results are usually not too far outside that interval, but you’ve got to draw a line somewhere.) Thus the “19 times out of 20” that frequently follows the +/- figure. In fact, that value is based on 50% of the population expressing whatever is being measured; the width of this interval becomes smaller as one approaches the extremes.
It’s possible to establish higher confidence levels, but 95% is by far the most widely used — it’s high enough to be useful, yet not so wide as to be overly fuzzy; a 99% confidence interval is about 30% wider than one of 95%.
What I’m more curious about is which polling stations were targeted by the survey, considering how widely the results varied across the city.
Following up on Andy’s comment, the NPA also squeezed Vision by buying up much of the available outdoor advertising within the city. This forced Vision to resort to print and broadcast advertising. A lot of the Vision ads were in the4 commuter tabs, which are concentrated in the city proper, so that wasn’t too bad for them. What was less efficient was having to run ads on the BCTV News Hour, where 3/4 or so of the audience can’t even vote in Vancouver.
COPE was handicapped by not having much of an ad budget compared to 2002; in this way, they were relegated to their pre-2002 roles.