After the shock over a closer mayor’s race than anyone expected has worn off, some might think that Lee Miringoff, master pollster for Marist Poll, would strike a conciliatory tone and admit he had it all wrong. Nope.
Miringoff, who loves going inside the numbers on elections the way others obsess over baseball statistics, explains how the polls accurately predicted Mayor Michael Bloomberg’s slim margin of victory, why Bill Thompson’s pollster Geoff Garin’s poll was way off, and what to do about cell phones and young people without landlines.
The following is an edited transcript.
City Hall: Why were the polls unable to predict the narrowness of Mayor Bloomberg’s victory?
Lee Miringoff: That was a very important point. I think there has been a tremendous amount of misinformation regarding the pre-election polls and the results of Election Day, which I must say we saw coming and tried to steer the media in the direction that this race was in fact going to get closer. Six days before the election, we released our final pre-election poll—and when you’re six days out from the moving event, it’s not designed to get precisely the margin.
When you have a well-known incumbent and a relatively less well-known challenger, as was in this case, the incumbent typically gets what he’s getting in the pre-election polls. What we saw was that in all the pre-election polls save one, I think, Bloomberg was always getting 51, 52, 53 [percent], which was very close to what he ultimately got. What that means from an accurate interpretation of poll results, not the horse race margin but the actual interpretation that you need, is that almost half of the city electorate was saying, “I’m voting for the other guy,” or, “I’m not sure.”
What you don’t want to do is focus on the margin. Let me give you another important example of why Bloomberg didn’t win in a landslide, which he was never going to do. The Bloomberg people understood that. The Thompson people understood that. The people doing the pre-election polls understood that. The problem was that most of the electorate, about 80 percent, thought Bloomberg was going to win. So the feeling was in from the beginning, and that created an atmosphere that this thing was done from day one—and that wasn’t really the case.
CH: So if a lot of the problem rests with how the polls are being interpreted, how do we change the way that polls are discussed in the media and the way the information is disseminated?
LM: I think it’s important to include in the interpretation of the polls what the interpretation is. It was done in some instances to the point of beating a dead horse a week before the election, and going into the election on Monday and then Tuesday. Every time I talked to someone, I said that this race was going to get closer and that’s all you can do from a polling standpoint. It’s very hard if the focus is on the margin and Bloomberg is up by 12 points. You’re trying to look at a moving target and the pollsters have to try to communicate the best that they can and the media has to try to provide that context. It can’t just be the horse race and it can’t just be the margin.
CH: Did any of the post-election commentary focusing on how the polls were wrong strike you as similar to reactions last year with New Hampshire and Hillary Clinton?
LM: We were up in New Hampshire at the time and on the ground, almost all the reputable polls ended in the weekend. Monday morning was the time that Hillary welled up over what was going on and there was the suspense of how this race needed to continue. … Polls are estimates of where things are, and I think the numbers look very precise and scientific, but if you do polls or you’re a close poll-watcher as a journalist, you understand that these are ranges and moving targets at a point in time. In an ideal world, the media would look at the track record of the poll, how the people were selected and the quality of the interviewing. It’s probably a little too care-free to pick every number from every poll and just say they’re the same. Everyone knows that not every doctor is as good at providing diagnosis and not every car provides the same gas mileage.
CH: Thompson pollster Geoff Garin seemed to have had the closest numbers, about a six- to eight-point margin.
LM: Oddly enough, Garin’s poll, which obviously was sponsored by the Thompson campaign, was, believe it or not, way off on the numbers each candidate got. They had Bloomberg in the mid-40s and they had Thompson in the high-30s. When you look at the point spread, it’s almost like reporting the score of a baseball game without telling you what both sides got. What you have in this poll, ironically, which is being touted as having the moral victory of calling the race in terms of the margin, missed Bloomberg by around 5 points and was close to 10 points off on the Thompson number. Now that’s what you’re measuring. You’re measuring Bloomberg, you’re measuring Thompson, and you had an undecided vote of 17 points. All of the other polls actually nailed Bloomberg’s number pretty close, and the Thompson number comes from the late action that was moving his way. Not to sound totally contrary, ironically, but from a poll interpretation, the Garin poll was actually the farthest off, not the closest.
CH: As we become a more wireless culture, what needs to change about the technology of polling going forward?
LM: It’s a huge challenge and we are trying to deal with it just like many other organizations. Most people are including cell phones in their samples. Again, for the organizations that aren’t doing this, that’s something the media should note. The problem from a polling standpoint is a much more inefficient process to include cell phones. There’s no master list of cell phones. There’s a lot of things you have to ask people on a cell phone like, “Are you using heavy equipment or not?” You can’t use auto-pilots. Your yield on a cell phone is much more reduced. You’re addressing a non-coverage bias because there are people who we are missing potentially. In the 2008 presidential campaign there was a huge concern because of the cell phone issue and the youth vote.
There was a sense that Obama was going to do very well with the youth vote and they were going to be undercounting him. It turned out it wasn’t the case because young folks who had cell phones and young folks who had landlines didn’t behave differently in the voting booth. There’s no guarantee that such an example will continue indefinitely; in fact there will be more of a cell phone issue and less of a landline issue. There’s a way of addressing it, and so far it hasn’t been the sort of macro distortion that some people have feared.
Certainly, that wasn’t going on with anything related to the mayoralty election. It played pretty true to form as far as the way things were breaking down in the numbers. In four to five years from now, maybe even sooner, there’s going to be a more mixed-mode ways of collecting data. You should know that on Sunday and Monday, in the poll that we could not release or chose not to release, we actually did mixed-mode interviewing in both New Jersey and New York City, which included the traditional telephone survey, cell phones and interactive voice recording (IVR).
CH: Why wasn’t that released?
LM: I’m glad you asked. Not everyone does. [Laughter] The reason we don’t release it is just from a confusion standpoint, this is one place where polling can be confusing. If you put out some numbers on Tuesday, Election Day, based on your Monday night election polls, it will be the most accurate of all your polls because it’s closest to the actual event. If I put a poll that said Bloomberg was up 8 points, that’s not a good thing to be putting out on Election Day.
CH: So is being a pollster a good job or not?
LM: The whole polling thing is a very, very misunderstood process. It’s the farthest thing from picking up the phone, interviewing 1,000 people and figuring out what percentage each candidate is getting; it’s light years away from that. There are so many nuances into this. The amazing thing is not that polls by different organizations vary; the amazing thing is how often they’re similar. That’s why I was struck by the Marist, Quinnipiac and the Survey USA polls [that] repeatedly had Bloomberg in the low 50s in most of their clockings, and it really didn’t look like Bloomberg was locked in. We were all showing very similar results and I thought that was interesting given that there were three different organizations showing that. To me, it was actually the opposite conclusion when I saw that Bloomberg got 50, 51 percent. To me, that was very confirming. I’m sure I was in for some hard interviews but I spoke at great length six days before the election at City Hall and really cautioned them to watch out for this… and it was going to get close. I was asked if Thompson could win and it wasn’t really our place to call that. I would have been very surprised if this race didn’t end up in single digits.
Miringoff, who loves going inside the numbers on elections the way others obsess over baseball statistics, explains how the polls accurately predicted Mayor Michael Bloomberg’s slim margin of victory, why Bill Thompson’s pollster Geoff Garin’s poll was way off, and what to do about cell phones and young people without landlines.
The following is an edited transcript.
City Hall: Why were the polls unable to predict the narrowness of Mayor Bloomberg’s victory?
Lee Miringoff: That was a very important point. I think there has been a tremendous amount of misinformation regarding the pre-election polls and the results of Election Day, which I must say we saw coming and tried to steer the media in the direction that this race was in fact going to get closer. Six days before the election, we released our final pre-election poll—and when you’re six days out from the moving event, it’s not designed to get precisely the margin.
When you have a well-known incumbent and a relatively less well-known challenger, as was in this case, the incumbent typically gets what he’s getting in the pre-election polls. What we saw was that in all the pre-election polls save one, I think, Bloomberg was always getting 51, 52, 53 [percent], which was very close to what he ultimately got. What that means from an accurate interpretation of poll results, not the horse race margin but the actual interpretation that you need, is that almost half of the city electorate was saying, “I’m voting for the other guy,” or, “I’m not sure.”
What you don’t want to do is focus on the margin. Let me give you another important example of why Bloomberg didn’t win in a landslide, which he was never going to do. The Bloomberg people understood that. The Thompson people understood that. The people doing the pre-election polls understood that. The problem was that most of the electorate, about 80 percent, thought Bloomberg was going to win. So the feeling was in from the beginning, and that created an atmosphere that this thing was done from day one—and that wasn’t really the case.
CH: So if a lot of the problem rests with how the polls are being interpreted, how do we change the way that polls are discussed in the media and the way the information is disseminated?
LM: I think it’s important to include in the interpretation of the polls what the interpretation is. It was done in some instances to the point of beating a dead horse a week before the election, and going into the election on Monday and then Tuesday. Every time I talked to someone, I said that this race was going to get closer and that’s all you can do from a polling standpoint. It’s very hard if the focus is on the margin and Bloomberg is up by 12 points. You’re trying to look at a moving target and the pollsters have to try to communicate the best that they can and the media has to try to provide that context. It can’t just be the horse race and it can’t just be the margin.
CH: Did any of the post-election commentary focusing on how the polls were wrong strike you as similar to reactions last year with New Hampshire and Hillary Clinton?
LM: We were up in New Hampshire at the time and on the ground, almost all the reputable polls ended in the weekend. Monday morning was the time that Hillary welled up over what was going on and there was the suspense of how this race needed to continue. … Polls are estimates of where things are, and I think the numbers look very precise and scientific, but if you do polls or you’re a close poll-watcher as a journalist, you understand that these are ranges and moving targets at a point in time. In an ideal world, the media would look at the track record of the poll, how the people were selected and the quality of the interviewing. It’s probably a little too care-free to pick every number from every poll and just say they’re the same. Everyone knows that not every doctor is as good at providing diagnosis and not every car provides the same gas mileage.
CH: Thompson pollster Geoff Garin seemed to have had the closest numbers, about a six- to eight-point margin.
LM: Oddly enough, Garin’s poll, which obviously was sponsored by the Thompson campaign, was, believe it or not, way off on the numbers each candidate got. They had Bloomberg in the mid-40s and they had Thompson in the high-30s. When you look at the point spread, it’s almost like reporting the score of a baseball game without telling you what both sides got. What you have in this poll, ironically, which is being touted as having the moral victory of calling the race in terms of the margin, missed Bloomberg by around 5 points and was close to 10 points off on the Thompson number. Now that’s what you’re measuring. You’re measuring Bloomberg, you’re measuring Thompson, and you had an undecided vote of 17 points. All of the other polls actually nailed Bloomberg’s number pretty close, and the Thompson number comes from the late action that was moving his way. Not to sound totally contrary, ironically, but from a poll interpretation, the Garin poll was actually the farthest off, not the closest.
CH: As we become a more wireless culture, what needs to change about the technology of polling going forward?
LM: It’s a huge challenge and we are trying to deal with it just like many other organizations. Most people are including cell phones in their samples. Again, for the organizations that aren’t doing this, that’s something the media should note. The problem from a polling standpoint is a much more inefficient process to include cell phones. There’s no master list of cell phones. There’s a lot of things you have to ask people on a cell phone like, “Are you using heavy equipment or not?” You can’t use auto-pilots. Your yield on a cell phone is much more reduced. You’re addressing a non-coverage bias because there are people who we are missing potentially. In the 2008 presidential campaign there was a huge concern because of the cell phone issue and the youth vote.
There was a sense that Obama was going to do very well with the youth vote and they were going to be undercounting him. It turned out it wasn’t the case because young folks who had cell phones and young folks who had landlines didn’t behave differently in the voting booth. There’s no guarantee that such an example will continue indefinitely; in fact there will be more of a cell phone issue and less of a landline issue. There’s a way of addressing it, and so far it hasn’t been the sort of macro distortion that some people have feared.
Certainly, that wasn’t going on with anything related to the mayoralty election. It played pretty true to form as far as the way things were breaking down in the numbers. In four to five years from now, maybe even sooner, there’s going to be a more mixed-mode ways of collecting data. You should know that on Sunday and Monday, in the poll that we could not release or chose not to release, we actually did mixed-mode interviewing in both New Jersey and New York City, which included the traditional telephone survey, cell phones and interactive voice recording (IVR).
CH: Why wasn’t that released?
LM: I’m glad you asked. Not everyone does. [Laughter] The reason we don’t release it is just from a confusion standpoint, this is one place where polling can be confusing. If you put out some numbers on Tuesday, Election Day, based on your Monday night election polls, it will be the most accurate of all your polls because it’s closest to the actual event. If I put a poll that said Bloomberg was up 8 points, that’s not a good thing to be putting out on Election Day.
CH: So is being a pollster a good job or not?
LM: The whole polling thing is a very, very misunderstood process. It’s the farthest thing from picking up the phone, interviewing 1,000 people and figuring out what percentage each candidate is getting; it’s light years away from that. There are so many nuances into this. The amazing thing is not that polls by different organizations vary; the amazing thing is how often they’re similar. That’s why I was struck by the Marist, Quinnipiac and the Survey USA polls [that] repeatedly had Bloomberg in the low 50s in most of their clockings, and it really didn’t look like Bloomberg was locked in. We were all showing very similar results and I thought that was interesting given that there were three different organizations showing that. To me, it was actually the opposite conclusion when I saw that Bloomberg got 50, 51 percent. To me, that was very confirming. I’m sure I was in for some hard interviews but I spoke at great length six days before the election at City Hall and really cautioned them to watch out for this… and it was going to get close. I was asked if Thompson could win and it wasn’t really our place to call that. I would have been very surprised if this race didn’t end up in single digits.