Grading the Polls

A subplot of the 2012 election cycle was the pollster battle: GOP number crunchers showed Republicans competitive, Democrats and independent poll-takes showed Dems ahead. Ultimately, Republicans were wrong.

Before the votes were tallied Tuesday, Republicans talked about a 2004 turnout model. They said that Obama’s lustre had faded, and a poor economy would depress voter turnout among Democrats, young people and minorities well below 2008 levels. They said polls that incorporated the 2008 model were off.

Big national firms like Gallup and Rasmussen, for much of the cycle, made similar assumptions.

“The biggest charge was that we were oversampling Democrats,” said Terry Madonna, Director of the Franklin & Marshall College Poll. “But if you’re going to talk about oversampling, you have to talk about screening questions to get the likely voters. You have to talk about weighting adjustments, not just for party but for age and gender and whatever else you choose to weight.”

Franklin and Marshall’s final 2012 poll was 1 point off of the presidential race and dead on accurate with the margin for U.S. Senate.

“We’ve not had a history of having either party’s candidate outside the sample error of our poll,” Madonna said.

In public and in private phone calls, Republican operatives insisted over and over that independent polls were wrong; they were oversampling Democrats or undersampling men, etc.

Were they lying? Were they trying to create a false narrative in the hope that it would become a self-fulfilling prophesy? Democrats repeatedly accused SP&R of misleading.

Based on the shock Republicans showed Tuesday night, and more importantly on the trust I’ve established covering these operatives for the past two years, I think the answer is no.

“Republican pollsters were really wrong,” said one statewide GOP operative. “None of us ever believed they would turn out voters like they did. My hat’s off to them.”

Polling is science and math, but also instinct and art. So GOP pollsters sampled voters who were older and whiter than the average person who went to the polls on Tuesday, based on the belief that 2008 was a one-time outlier. They were wrong.

“They thought they were looking at a different electorate than the normal methods showed,” said Chris Borick, Director of the Muhlenberg College Institute of Public Opinion. “You don’t build a turnout model based on optimism or pessimism. I don’t even like to weight for party.”

Does he think they were deliberately misleading? Pause.

“Republicans use polls – as Democrats often do – for tactical reasons. I think the goals of their polling had an impact on the methods.”

Here are poll results from the home stretch, the final 2 weeks of the campaign and how their margins compare to the actual results. Barack Obama beat Mitt Romney 52 percent to 47 percent, a 5 point edge for Obama. From Real Clear Politics:

In the U.S. Senate race, Sen. Bob Casey beat Tom Smith 54 percent to 45, a lead of 9 points. Here are the final poll results in that race, also from Real Clear Politics:

Susquehanna Polling and Research conducted polls for dozens of Republican candidates in Pa. this cycle, as well as the Pa. Republican Party. On Oct. 18, they released a poll showing Romney ahead of Obama by 4 points in the state. Their polls for the Pittsburgh Tribune-Review, included in the RCP lists above, showed both races far more competitive than they turned out to be.

From today’s Tribune-Review:

Susquehanna underestimated turnout among Democratic-friendly segments of the population, said Susquehanna’s president, Jim Lee. Young voters and Philadelphia-area Democrats turned out in numbers close to those in 2008, when Obama won the state by 10 points, Lee said.

“That caught me off guard. I didn’t think that was going to happen. I didn’t think the enthusiasm was there,” Lee said.

Voters ages 18 to 29 were a key constituency in Obama’s 2008 victory, when they turned out in record numbers.

“I was suggesting it would be lower with that age group, given unemployment was considerably higher” for them, Lee said.

Here are some other polls of various Pa. races and how they stacked up against the results.

Presidential:

Pulse Opinion Research (for conservative non-profit Let Freedom Ring)
Oct. 30: Obama 49, Romney 46
Off by 2

U.S. Senate:

McLaughlin Group (Smith campaign pollster)
Nov. 2: Casey 46, Smith 46.
Off by 9

Pulse Opinion Research (for conservative non-profit Let Freedom Ring)
Oct. 30: Casey 46, Smith 45
Off by 8

Citizens United (conservative group)
Oct. 24: Casey 46, Smith 45.
Off by 8

Democratic Senatorial Campaign Committee
Oct. 26: Casey 52, Smith 40
Off by 3

PA-12. Final result: Keith Rothfus 51.5, Rep. Mark Critz 48.5. Rothfus by 3

Public Opinion Strategies (for GOP super PAC)
Oct. 8: Rothfus 42, Critz 40
Off by 1 (in Critz’s favor

Anzalone Liszt Research (for DCCC)
Sept. 26: Critz 52, Rothfus 41
Off by 14
Note: This poll was taken 6 weeks before the election, so events of the campaign could account for the difference just as easily as anything else.

SD-37. Final result: Matt Smith 52, Raja 47. Smith by 5

39th Street Strategies (for Smith campaign)
Oct. 25: Smith 53, Raja 40
Off by 8

SD-49. Final result: Sean Wiley 60.5, Janet Anderson 39.5. Wiley by 21

39th Street Strategies (for Wiley campaign)
Oct. 24: Wiley 57, Anderson 38
Off by 2 (in Anderson’s favor)

HD-83. Final Result: Rep. Rick Mirabito 59, Harry Rogers 41

Lycoming College (independent poll)
Oct. 29: Mirabito 59, Rogers 31
Off by 10
Looks like undecideds went to the challenger in the GOP-friendly district.

11 Responses

  1. Old Man Dickie Mellon Scafe pouring money into “polling” that is nothing more than laugh out loud propaganda. No one was buying that garbage when the data was released because even putting Susquehanna on the byline can’t hide the source, who while on his death bed still consistently delivers a racist woman hating agenda advanced by his halfwit minions in the editorial dumpster. Note how the minions bust on Susquehanna today in an attempt to (toilet) paper over the purchase of worthless poll data. That is priceless! Which begs the quesiton is there nothing poor adopted Dickie’s money can’t buy? Yes and that’s credibility and respect!

  2. Adam and David, all pollsters estimate turnout. Some pollsters even issue a caveat that the accuracy of their results depend on turnout. The part of the poll that reflects the voters’ preferences may be scientific, but predicting turnout is more of an art. The standard practice is to use the previous election’s turnout as the model to predict the current one.

    The Republican pollsters, like Susquehanna, were correct that, Democratic turnout would be less than in 2008 in Pennsylvania, and especially across the United States, although not as less as predicted, while the other pollsters who predicted the Democratic turnout would match or even exceed 2008 were wrong. However, Republican turnout was also down slightly, which was not predicted by the pollsters. The election results were close both in PA and across the U.S. because in part because the decrease in Democratic turnout was larger than the drop in GOP turnout. Therefore, none of the pollsters accurately estimated the turnout for either party, although none of them were as way off as the author implies, either in estimating turnout or the result. They were all close.

    Susquehanna was also accurate that Republicans would make gains versus Obama in the Philadelphia suburbs and western PA, which was another major reason the result was much tighter than 2008.

  3. Let’s be clear though, the Susquehanna polling was based on 2004 turnout levels–and state turnout was below that. And it wasn’t just on the Republican side – Obama got fewer votes not only than in 2008, but than Kerry(!) got in 2004.

    And David – yes, Romney didn’t have the as great a “ground game” as Obama. Everyone knew that was going to be the case. But he got few votes than McCain did in 2008 – and McCain had absolutely zero ground game and ran an awful campaign. Romney and conservative groups put a lot more into GOTV efforts than in prior years.

    The failure of this massive GOTV effort (on both sides) should be the big story for campaign pundits.

  4. Guessing enthusiasm?

    Sounds like another way of saying they pulled the numbers from their rear orifices.

  5. Susquehanna’s explanation that he guessed wrong about enthusiasm made me laugh. YOU’RE A POLLSTER! DON’T GUESS AT ENTHUSIASM, ASK THE VOTERS!!!!

  6. Nathan-
    The Romney campaign certainly didn’t have the field offices and ground game that Obama had.

    Either they knew the state was unwinnable, or were counting on voterID to suppress votes.

  7. That explanation makes no sense – turnout was way down in Philadelphia and statewide from 2008 totals. It was even lower than in 2004! If anything explains the bad polls, it was a lack of turnout by Republicans (even independents), not a big turnout by Democrats.

  8. If we really believe in “counting every vote,” then we should wait until the absentee and provisional ballots are counted before reporting “Final Results.” As Republicans usually do significantly better in absentee votes, in particular, the numbers will probably tighten somewhat.

    Nonetheless, all the presidential pollsters, including the Republican ones, accurately had the race as a statistical dead heat, within the margin of error. Most people in the media, liberal or conservative, apparently are unaware that the margin is plus or minus for each figure in a poll, not the difference between them. See also http://williamcinfici.blogspot.com/2012/09/the-medias-error-on-margin-of-error-of.html.

Email:
  • Will tonight's U.S. Senate debate affect your decision?


    • No. I've already decided on how to cast my vote. (81%)
    • Yes. Anxious to hear from both candidates (19%)

    Total Voters: 27

    Loading ... Loading ...
Continue to Browser

PoliticsPA

To install tap and choose
Add to Home Screen