Evaluating the Polls: an Open Letter to Ontario’s Journalists
Wednesday, September 14, 2011
Toronto, ON – We’ve all seen over the last few days a confusing cacophony of polls on the Ontario election. Depending on what poll you read, McGuinty's Liberals are on a roll, Hudak and the Tories are comfortably ahead, or the Grits and Tories are neck and neck. How can this be? It is because all polls are NOT created equally. And, in spite of what you may assume, pollsters are never held to account for their indiscretions, incompetence and mistakes (there is no “polling police”). Some marginal pollsters count on your ignorance and hunger to make the news to peddle an inferior product. Others are using your coverage to "prove" that their untried methodology is the way forward for market research in Canada. Instead of being their own biggest sceptics (which is what our training tells us to be), they've become hucksters selling methodological snake oil. Remember, the term "pollster" is derived from the term "huckster".
Journalists are no mere dupes in this process. We've also seen a disturbing trend of late in which questionable polls find their way into an outlet’s coverage because they appear to match an editorial line, or present a counter-intuitive perspective. After all, if a poll is wrong it’s easy to throw the pollster under the bus and walk away with clean hands.
All of this MUST stop. We are distorting our democracy, confusing voters, and destroying what should be a source of truth in election campaigns - the unbiased, truly scientific public opinion poll.
To be clear, this is not about banning media polls during election campaigns. That would just take us back to the old days of backroom boys leaking false polling, and to the practices we see in less stable democracies around the world. What we need is better, more informed reporting of polls. Here are six easy rules to get us started.
1. IVR polls (robo-calling) are NOT telephone polls. They are tremendously biased in terms of sample coverage. In the last federal election IVR polls were MASSIVELY off on the final vote. Why? Because they systematically under-represented the Tory vote. But, these same pollsters have picked up again in this election without skipping a beat. Here's the question you should be asking them - show us your UNWEIGHTED results. That's prior to adjusting for both demographics and political support. You'll be surprised by what you see.
2. The same question should be asked of pollsters using on-line methodologies to predict vote. Ask to see their results prior to all weighting. You will find that some heavy thumbs are being applied to adjust for under-represented voting groups. While the weighting can produce very good results, it really amounts to no more than an educated guess. And, if that's the case, the results should be reported as such.
3. Disclosing margin of error and the questions asked doesn't represent meaningful disclosure: a rogue poll, a bad poll, and a good poll all look the same on these points. Be honest when something looks dodgy - either don't publish it, or publish it with an editorial disclaimer. Huge shifts in public opinion, even in an election, are the exception, not the rule and should be treated as dubious until confirmed by other polls.
4. A moratorium should be placed on all "new" polling techniques until the pollster has tested them in parallel with more traditional polling methods with a record of success. This should have absolutely been the case with IVR in the federal election. It's a pretty minimum standard to be right at least once with a new methodology before you get to lead a newscast or get the front page headline in a newspaper.
5. The conventional election scenario that's tested in a vote question is - if you had to vote tomorrow, how would you vote? If your pollster asked something different, even if the poll was properly done, it should NOT be reported as "current vote". It is misleading to do so.
6. Spend some time with pollsters. Not just the one you know or use, but with the ones you're hearing about. They are NOT all created equal. Find out about their business. Is this a one person show that subcontracts all of their data collection, and only shows up at election time? What research are they doing on research? What resources do they have available - is it one way for everything, or do they have the resources to adjust depending on what's needed? What are their motives? Are they just publicity hounds trying to promote their business, or are they serious researchers with both the desire and capabilities to perform at the highest level? Shouldn’t you do at least this minimal background check on whomever you are trusting with your lead story?
These six rules require journalists to do two things - kick the tires before publishing a poll, and make it harder for bad or misleading polls to get published. That's the way it's done in jurisdictions that take polling seriously. It's sad that this isn't the case in Ontario today.
Darrell Bricker and John Wright lead all of Ipsos Reid’s political polling for the media in Canada. They have done polls in every major election in Canada since 1988. Ipsos Reid is Canada’s largest market research firm, and is owned by Ipsos, the world’s third largest market research firm.
For more information on this news release, please contact:
Ipsos Public Affairs
SVP & Managing Director, Public Opinion Polling
About Ipsos Reid
Ipsos Reid is Canada's market intelligence leader, the country's leading provider of public opinion research, and research partner for loyalty and forecasting and modelling insights. With operations in eight cities, Ipsos Reid employs more than 600 research professionals and support staff in Canada. The company has the biggest network of telephone call centres in the country, as well as the largest pre-recruited household and online panels. Ipsos Reid's marketing research and public affairs practices offer the premier suite of research vehicles in Canada, all of which provide clients with actionable and relevant information. Staffed with seasoned research consultants with extensive industry-specific backgrounds, Ipsos Reid offers syndicated information or custom solutions across key sectors of the Canadian economy, including consumer packaged goods, financial services, automotive, retail, and technology & telecommunications. Ipsos Reid is an Ipsos company, a leading global survey-based market research group.
To learn more, please visit www.ipsos.ca.
Ipsos is a leading global survey-based market research company, owned and managed by research professionals. Ipsos helps interpret, simulate, and anticipate the needs and responses of consumers, customers, and citizens around the world.
Member companies assess market potential and interpret market trends. They develop and build brands. They help clients build long-term relationships with their customers. They test advertising and study audience
responses to various media. They measure public opinion around the globe.
Ipsos member companies offer expertise in advertising, customer loyalty, marketing, media, and public affairs research, as well as forecasting, modeling, and consulting. Ipsos has a full line of custom, syndicated,
omnibus, panel, and online research products and services, guided by industry experts and bolstered by advanced analytics and methodologies. The company was founded in 1975 and has been publicly traded
since 1999. In 2010, Ipsos generated global revenues of €1.140 billion ($1.6 billion U.S.).
Visit www.ipsos-na.com to learn more about Ipsos
offerings and capabilities.
Ipsos, listed on the Eurolist of Euronext – Comp B, is part of SBF 120 and the Mid-100 Index, adheres to the Next Prime segment and is eligible to the Deferred Settlement System. Isin FR0000073298, Reuters ISOS.PA, Bloomberg IPS:FP