12 November 2012

There's bias and then there's bias

Rhymes With Cars & Girls | Sonic Charmer | Nate Silver finds and quantifies evidence of (D)-favoring voter fraud

I wish this was true. I really do. But Charmer said people should call him out more often, so that's exactly what I'm going to do.

As much as I wish I could embrace this thesis, I think Charmer is making a big mistake, and that's conflating bias and variance.
Nate Silver has an interesting post summarizing how the various polling firms did in their state polls. Generally it looks like he finds most of them had consistently overestimated the (R) vote. He throws out the usual cuffed explanation for this (not enough cell phones, younger people have cell phones, younger people are more (D), bla bla).

But he appears to miss the elephant in the room, which is cheating. But surely no serious, scientific, quantitative genius person such as Nate Silver can possibly forget or just un-scientifically ignore the fact that the final vote contains some nonzero amount of cheating.
So far, so good. Cheating accounts for some fraction of the discrepancy. A non-zero fraction, but also a non-one fraction. Even if all the polls are perfectly done, there will be some tiny, tiny correlation between them. Just like it's safe to assume at least one ballot was fraudulent, it's safe to assume at least the estimate of one vote did not cancel out when all the different polls were conducting and aggregated.
Now – to echo a bunch of arguments I made against Silverbating righties – the fact that almost all these polls from all these different polling companies with all different sorts of methodologies find a consistent, systematic “(R) bias” just beggars belief. No quantitative-minded person can just accept that as the result of random chance. Sure, there will be errors and biases but wouldn’t the errors and biases cancel each other out? How likely is it that virtually all polls would come out with an (R) bias? That is an extraordinary claim which requires extraordinary evidence, which Nate Silver does not have.
Here's where things get semantically tricky, because "bias" has a particular meaning which is different from the one we use when we talk about politics. At least it does according to the quantitatively-minded persons who taught me quantitative things.

If a system has bias, it will make errors on a particular prediction regardless of the training set (or in the case of polling, sample set) used. Variance, OTOH, are errors that will change for that particular prediction based on which training set/polling sample is selected. Bias is, roughly, a systematic error, while variance is a result of "noise."

For example, the C4.5 decision tree learning algorithm has a bias towards orthogonal classes. If the true decision surface isn't orthogonal in a particular dimension then it will always make some errors because of that. If the true decision surface is orthogonal but slightly amorphous, those will cause variance errors.


(The left is variance. If you're predicting blue o's will be on the left of x=0.5 and red x's on the right, you'll have gotten some wrong just based on the random noise. The right is bias. If you still thought x=0.5 was the dividing line between classes but it turns the true dividing line is the solid purple diagonal line then you're going to make consistent and correlated errors no matter what random sample of x's and o's you select.)

The errors of every predictive system are a combination of bias and variance. In order to avoid bias you need flexibility, but increased flexibility makes you subject to over-fitting. As a result you'll always have some of each kind of each error.

So it's totally possible, in fact likely, that each poll could have correlated bias. It's implausible they'll have correlated variance, but bias is extremely likely. If there is in fact problem with getting Dems on the phone, etc. and every poll is based on randomly phoning people, then yes, you would expect all those phone polls to make correlated errors.
The more parsimonious and scientific inference is that the “(R) bias” Nate Silver has found is, of course, nothing other than an estimate of the (D) cheating advantage. What else could it be, after all? Yes, it could theoretically be something else – but that would require an explanation, and evidence. Surely the null hypothesis is that the “bias” showing up from these polls is just the result of voter fraud.
Can we assume that both parties cheat to some extent? There is at least one pro-GOP and one pro-Dem ballot that has been cast fraudulently. In any set of elections, some of them will receive more fraudulent R than D votes. If each party were to cheat the same amount we would see these canceling each other out; there would be no correlation between the cheating levels.

Okay. In order for Charmer's hypothesis to be the most parsimonious, we must conclude that the correlation of cheating exceeds the correlation of poll errors.

I'm totally willing to believe one party cheats more than the other. Fine. But I also have good reason to believe that the poll errors are highly correlated. What I don't have is reason to believe ballot fraud is more highly skewed Dem than is polling error. We're right back where we started: some of the polls' overestimation of GOP votes is due to bias in the polls, and some is due to fraud. Still no reason to conclude the fraction is 100% fraud and 0% polling bias.
In other words, we now have scientific, quant-friendly evidence here that the (D)s get something like a ~1.2% advantage from fucking cheating.
This is true only if you assume all the differential is due to cheating. This is the ceiling of that the Dem advantage to cheating is; the true value is lower.

In order to figure that out we would need to know not only that one party cheats more than the other, but the degree to which they do so.
Again, if you have a better explanation why all these polls would come out with a R+1.2 bias on average, you are welcome to advance your argument, along with your evidence. But fair warning, if you mumble some facile BS about ‘cell phones’ or ‘hurricane Sandy’ I’m going to fucking make just as much fun of you as I made of righties who BS’ed stuff about lefties more likely to lie about being likely voters or pollsters ‘using a 2008 turnout model even though (R) enthusiasm is really high’.
Cell phones? Sandy? I don't know what the underlying cause is, if indeed there is a single one. But you don't have to be able to solve that responsibility assignment problem in order to know if there is bias.

If every poll is making the same assumption about the distribution of the electorate, for example, assuming there is no correlation between being willing to take 5 minutes to talk to a pollster and supporting Romeny, and that assumption is wrong then they could all very easily make correlated errors because they would have the same bias. Not the same variance, but the same bias. Since all predictive systems will make both bias and variance-derived errors, if the bias terms are correlated, the error terms will be correlated. (Less strongly, due to canceling variances, but still somewhat correlated.)
I think I’ve earned the right to say this because I have been and continue to be consistently on the side of the quants: if you don’t see Nate Silver’s table as, absent other quantified and supported explanations, prima facie evidence of the size of the (D) cheating advantage, then guess what? You’re not on the side of the quants, and you must hate math.
I think I've also earned the right to say this, because I have also consistently been on the same side of this issue. So... ummm... there. Take that?

PS If every poll's errors were uncorrelated, why would we see this?

If Charmer's story is correct, and polls are wrong only because of cheating, then it shouldn't matter what kind of poll was conducted. They'd all accurately reflect how people intended to vote, people would go out and vote that way, then the results would be skewed by fraud. All the poll types would show the same GOP overestimation/Dem cheating advantage.

This seems to be pretty clear evidence that the polling process itself introduces correlated errors.

PPS Internet polls most accurate? Interesting. Contradicts conventional wisdom. Would like to know more.

I'm editing the above to reflect a point Charmer (aka RWCG, which I'll call him (?) from now on) made in the comments.

Internet polls more closely reflect the observed outcome? (* Where the observations include effects of fraud.) Interesting. I would like to know more about their process.

13 comments:

  1. Thanks! I do need more criticism on this post of mine.

    I may have to chew on this a bit longer to make sure I understand it. For now,

    "So it's totally possible, in fact likely, that each poll could have correlated bias. It's implausible they'll have correlated variance, but bias is extremely likely."

    We presumed polls had no such correlated bias for the purpose of backing out Obama's chances to win. One of the reasons we did this was instrumental: because that way, we could get an answer that we could look at, whose movement we could watch, spark discussions, etc. But another reason (I claim) was that the proferred explanations of such correlated/systematic biases were unconvincing, post hoc just-so stories.

    I never bought into them. You didn't either. And so, parsimoniously, we decided that the best, most valid approach was to treat them as untrue - unless/until convincing reasons surfaced otherwise. Could such reasons exist? Sure! But no good, convincing evidence was ever given.

    All I am suggesting is that the same applies here. Yes, of course there could indeed be a systematic bias among The Pollsters that tilts them all (R) on average (+/- their own internal variance and slight methodology diffs etc). But there's no good reason to believe that at all. Some may think 'because the votes, once tallied, differed significantly from the average poll' is a good reason, but I am pointing out that to say that is to forget about or deliberately ignore that the 'actual tally' contains cheating.

    And of course the real purpose of my post was just to point out that cheating exists :)

    "This is true only if you assume all the differential is due to cheating. "

    But that's the only possibility left once we assume state polls have no systematic/correlated bias. Which we all had agreed to do, pre-election (I thought!) I mean, if that assumption isn't warranted, then Silver's calculations were all incorrect.

    "Can we assume that both parties cheat to some extent?"

    Of course. That's why I spoke of the (D) 'cheating advantage'. I never implied and do not believe that it's a question of one party cheating and the other party not cheating at all. The question is whether/to what extent one party has a systematic advantage in the cheating that they both do.

    Looks to me like the best current answer is 'yes, the Ds, by about 1.2% or so'.

    UNLESS, of course, some REAL evidence or argument is given to the contrary as to why those polls would lean that way. I don't deny the possibility of such evidence or explanations. I just think it is very weak at the moment - every bit as weak as was the righties' evidence/arguments for their pre-election claim that the polls were all D-skewed. I literally cannot distinguish a meaningful difference in quality or supporting evidence between the two types of arguments.

    Can you?


    best

    ReplyDelete
  2. P.S. Just a note: the table at the end of your post is *not* evidence that "Internet polls are most accurate". It is evidence that Internet polls matched the final tally the best.

    But the final tally contains cheating!

    This is precisely the point I was trying to drive home with my post, because I think a lot of people miss it.

    ReplyDelete
    Replies
    1. Excellent point. I'm updating my post to reflect that.

      Delete
  3. "But another reason (I claim) was that the proferred explanations of such correlated/systematic biases were unconvincing, post hoc just-so stories.

    I never bought into them. You didn't either. And so, parsimoniously, we decided that the best, most valid approach was to treat them as untrue - unless/until convincing reasons surfaced otherwise. Could such reasons exist? Sure! But no good, convincing evidence was ever given."


    I agree those stories are just that -- stories. But I don't think that means there is zero correlation between polls. Since all these polls are conducted roughly the same way, and they'll all have bias terms in their error, I would expect some non-zero correlation between their errors.

    The reason I wanted to ignore all those just-so stories is because the stories don't tell us anything about the size of the polling bias. I could concede every single story about young dems with cellphones and Bradley effects and on and on, but that doesn't give me any way to move from the reported poll results to a "real" result. I don'tknow if the effects in those stories are bigger or smaller than the leads the polls reported for Obama or Romney. I'm not even sure of the sign of the effects.

    So I ignore those stories not because I think the size of their effect is zero, but because I have no idea what the size of the effect is, so it doesn't do anyone any good to spill ink insisting what the "real" numbers should be.

    "But that's the only possibility left once we assume state polls have no systematic/correlated bias."

    But I don't think they do have no correlation. I insist it's positive. It might be infinitesimally small, in which case the true amount of cheating is effectively equal to the figure you gave, but that figure still represents the ceiling.

    "'Can we assume that both parties cheat to some extent?' Of course."That part was more for the rest of the audience than for you.

    "Looks to me like the best current answer is 'yes, the Ds, by about 1.2% or so'.

    UNLESS, of course, some REAL evidence or argument is given to the contrary as to why those polls would lean that way."


    I think this is where we disagree. Maybe this is my machine learning background, but I would assume that if you use roughly the same technique to make multiple predictions, those predictions will be correlated to some non-zero extent. I if I train a thousand multi-layer perceptrons on samples drawn from some training set then the errors they make won't be perfectly uncorrelated. If I conduct a thousand robocall polls from the registered voters of Ohio I don't think the results will be perfectly uncorrelated.

    I'm willing to dismiss all the post-hoc stories about where that bias comes from in the polls for the same reason I'm unconcerned about the bias in the MLPs: I don't care, and it doesn't matter to me unless you can first measure it and then fix it. It's a black box. I expect there to be a bias in there even if I don't expect us to be able to put our finger on the exact nature of it. We can scratch our heads and take guesses, but it's a complex system and our guesses are probably going to be too simple to be right or useful.

    At the end of the day I don't find a story which hinges on all 100% of the poll-vs-ballot difference to be due to cheating to be any more parsimonious than a story which requires all 100% of the difference to be due to polling bias. Either way you're assuming something which ought to exist to some unknown extent does not exist at all.

    ReplyDelete
  4. PS

    "I literally cannot distinguish a meaningful difference in quality or supporting evidence between the two types of arguments. Can you?"

    I think you were smart to identify these two arguments as being roughly equivalent.

    I would expect theoretically for there to be some bias in the polling process. I don't know the exact nature of it, nor could I know in advance the magnitude of it. So I was perfectly willing to ignore all the griping about it, especially from people who were more interested in giving their partisans pep talks than doing analysis.

    But I do think that "because the votes, once tallied, differed significantly from the average poll is a good reason" to believe the theory is true. I don't think extra evidence is needed. I don't think any extra evidence for cheating is needed either. I'm only claiming that the difference between poll predictions and observed outcomes is some *mix* of both factors.

    ReplyDelete
  5. "But I don't think that means there is zero correlation between polls."

    It doesn't 'mean' that there is zero correlation between polls, but it does provide a quanty, numerate, sciencey skeptic with reason to act as if there is zero correlation between polls (or I would rather phrase it, for reasons that become clear below, zero systematic poll bias on average) unless/until good arguments appear.

    Right? I mean, everyone was totally convinced of this before Tuesday. What changed?

    "I'm not even sure of the sign of the effects."

    Precisely. Hence, lacking other info, we are indifferent and assume the effect washes out. Or, we used to. I guess we suddenly don't now?

    "But I don't think they do have no correlation. I insist it's positive. "

    Okay, say I agree with you the correlation is positive. (And for the record, I have absolutely no idea whether all these polls use 'roughly the same method', but let's say they do.) But unless I'm missing something, 'positive correlation' can cut either way. Sure, maybe all the polls are correlated - and systematically favored to the Ds by 5%. Who knows? We would then infer the true cheating is D+6.2%, right?

    It's fine to say poll methods are 'correlated' but even if so we have absolutely no way of selecting the size or direction of that effect. Hence we parsimoniously assume zero systematic bias in polling and see what happens. What happens is:

    1. Obama was the clear favorite
    2. The final vote tally appears to contain some net cheating for the Ds

    It's possible and fine to reject both of these, but I don't see how it's possible to reject just one.

    "Either way you're assuming something which ought to exist to some unknown extent does not exist at all."

    Well, then this is an argument against Nate Silver's model. It 'assumed' that the state polls (properly weighted, whatever he meant by that) contained no systematic statistical bias. And you don't like that assumption, because of course (as you argue convincingly) a systematic statistical bias is to be expected if only due to correlation that comes from similarity in poll methodology.

    So, Nate Silver was....wrong after all?

    "I'm only claiming that the difference between poll predictions and observed outcomes is some *mix* of both factors."

    For the record, I of course agree with you that in reality it is 'some mix' of both factors. I am being somewhat snarky in insisting that the mix is 100% cheating; I had a particular point to make.

    best

    ReplyDelete
  6. Before the election there was no credible evidence that the polls were biased in one particular direction or the other. The rational way to use the polls to make predictions was to assume zero bias, or if you wanted to be more complicated, bias with some normal probability distribution centered on zero.

    After the election there was no credible evidence of fraud favoring one party over the other. However, the discrepancy between the polls and the outcome is EITHER a) evidence of bias given zero differential fraud, OR b) evidence of differential fraud given zero bias.

    Which of the two is more credible - or rather, their relative credibility - completely depends on your priors for the existence and magnitude of bias and fraud.

    My priors are that fraud is extant but small, and that bias is extant and unknown (in magnitude and sign) and quite possibly very large. The discrepancy between polls and votes thus leads me to believe that poll bias is roughly R+1.2, plus or minus the small amount my priors tell me could be due to cheating.

    Your priors may lead you to different conclusions.

    What are your priors?

    ReplyDelete
    Replies
    1. eddie has articulated better than I how to think about it.

      My point with the post was that the election tallies don't - despite how Silver etc. have greeted them - really give anyone reason to update their pre-election priors about poll-bias being zero, *unless* they also have a strong prior involving vote-cheating being 'small'.

      This is not to say that I really think cheating necessarily accounts for the entire 1.2, but it is nevertheless true that I have no such prior about cheating's effect being 'small'. And that YMMV, obviously.

      best

      Delete
    2. I'll also add: it's inconsistent to say pre-election that "the polls are unbiased because they are uncorrelated" and post-election that "the polls are biased because they are correlated". This is a separate point worth teasing apart from the other points. I think this is one of your main points, and you're right.

      However, that said, that's probably not a fair characterization of anyone's actual viewpoint, even if that's how it seems to come out when they talk about it. I think a reasonable viewpoint, and the one which Nate and SB7 probably hold, is this:

      Pre-election position: there is probably some bias in the polls, but it is probably small, and certainly unknown in magnitude or sign. The fact that the polls are independent should make them at least kind-of uncorrelated, which should keep the bias small. And anyway, even if they were correlated and had a significant bias they still give Obama a huge probability of winning the election because math so NEENER NEENER NEENER you innumerate slobs.

      Granted, in the case of SB7 there's the tacked-on ".. alas. Sigh."

      Post-election position: Yeah, we said the polls were independent and uncorrelated... MOSTLY, PROBABLY. And now because of the discrepancy we can see that, yep, they were, MOSTLY, but not perfectly, which means that they are actually correlated, A LITTLE BIT, KIND-OF, to the tune of, oh, about 1.2 percent bias.

      It's not the positions that have changed, only the emphasis from MOSTLY no to KIND-OF yes.

      Delete
    3. You two are keeping me on my toes so much I'm having a hard time keeping up with the comments on my own post.

      Eddie, I think you summed it up very well. My position has changed from "mostly no (at least not enough to allow us to draw other conclusions about the outcome and then bitch about them heavily on talk radio)" to "kind of yes (in light of observed results, but with the caveat that some of the discrepancy is fraud)." Is that consistent?

      PS RWCG says "Sure, maybe all the polls are correlated - and systematically favored to the Ds by 5%. Who knows? We would then infer the true cheating is D+6.2%, right?" It has occurred to me that perhaps the linear combination of polling bias and fraud was not convex, but I forgot to put that in one of my replies. This is a good point to raise, but this takes us even further away from parsimonious explanations, right?

      Delete
    4. One other thing:

      RWCG also said "Okay, say I agree with you the correlation is positive. (And for the record, I have absolutely no idea whether all these polls use 'roughly the same method', but let's say they do.)"

      Do you two think the table I posted that shows different discrepancies between the final tally and live calls, robocalls and internet polls is some evidence that polls (at least within the same type) are correlated? Why would cheating affect one more than the others? Or is there too little data to tell?

      I suppose I could set up a simulation to find out, but I've already spent enough time on this. What's your intuition: evidence of correlation between polls, or natural variation in a small sample?

      Delete
  7. aire bra
    To leave the world a bit better, whether by a healthy child, a garden patch
    or a redeemed social condition;

    ReplyDelete
  8. Oh, I definitely agree that (-5,+6.2) is much less parsimonious than my actual inference, (0,+1.2). :-)

    Re: correlation, on reflection I think the focus on this notion of correlation is actually a red herring that needs to both be specified more carefully, and separated out from systematic bias a bit better. What we care about (and need to have a prior about) is whether the polls 'in general' have a bias, not with whether they (or more precisely, their sampling etc. errors) are correlated. The former can be true with or without the latter, and vice versa:

    Bias without correlation: all poll companies construct perfect, random, mean-0-error polls, but secretly add 1% to the R total before publishing.

    Correlation without bias: poll company #1 constructs a perfect, random mean-0-error poll, and companies #2-n just secretly steal and republish the results of #1.

    The sub-table showing different means for internet polls, cellphone polls etc is interesting but we still appear to have the overall pattern of a bias to the Rs. So either way, the results give us a reason to believe that the polls in general were not unbiased estimators of the final tally, but (perhaps?) were unbiased estimators of the final tally + 1.2R. (Note: of course I'm too lazy to do any real statistical checks on any of this. :) This is the observed 'mean-of-means' regardless of the issue of correlation in poll errors.

    So the question remains how we interpret that tally = polls + 1.2R result, for which, see eddie's comment above.

    ReplyDelete