Poor Statistical Reasoning On Last Week Tonight?

Image result for john oliver vaccines

On a recent episode of John Oliver’s Last Week Tonight, Oliver discussed the growing problem of parents choosing not to vaccinate their children (watch for free here) . It’s an interesting episode in its own right, but especially interesting is the conversation that starts at 12:00, where the discussion turns to whether the mercury in vaccines is harmful to people receiving the vaccine. In what follows, I point to what I believe to be a flaw in Oliver’s (and a guest scientist’s) reasoning about the absence of a link between mercury in vaccines and autism. It’s important to note that this criticism is in no way an endorsement of the idea that such a link exists.

To study whether the mercury in vaccines is harmful to people receiving the vaccine, we can, of course, imagine a well-crafted clinical trial designed to test whether mercury is harmful. Such a study would have a null hypothesis that “mercury in vaccines is not linked to X”, where X might be autism in infants, or some other harm to humans receiving the vaccine.

Whenever a scientific claim is being testing using statistical significance testing (as many are), the null hypothesis is assumed, and calculations, e.g., of a p-value, are done under that assumption. Consequently, when one fails to reject the null hypothesis, logically, one cannot claim that the null hypothesis is true; further, whether failing to reject the null hypothesis constitutes evidence that the null hypothesis is true is difficult to assess. (Reasonable people disagree on this point.  See the debate about inductive support vs. corroboration here.) Strictly speaking, all that one can say is that one has not found evidence against the null hypothesis.

This creates a problem for Oliver and researchers of vaccines, which is pointed out by Rep. Dan Burton. Burton makes a claim about our ability to prove (or obtain evidence in favor of) “null hypotheses”. Burton and Oliver couch this in terms of “proving a negative” (which, as this paper points out, isn’t really the correct characterization of the problem).

“I have yet to find any scientist who will say that there’s no doubt that the mercury in vaccines does not contribute to autism.  Now they’ll say that there’s no scientific evidence, there’s no studies or anything that proves that yet. But turn that around. There are no studies that disprove it either.”

Oliver attacks Burton pretty harshly. More eloquently, Seth Mnookin describes the way that scientists purportedly reach conclusions about the truth of null hypothesis here:

“Science and English are not really the same language. And so, when a scientist says we have no evidence that there’s a link between vaccines and autism, what they’re really saying is we are as positive as one can humanly be that there’s no link.”

This exchange, in conjunction with what we outlined about hypothesis testing above, leads to an interesting problem. If null hypotheses are assumed, then failing to reject a null hypothesis doesn’t seem to give evidence for the null. So, on what basis are scientists “as positive as one can humanly be that there is no link”?

I’m not sure. But, I have a hypothesis. Many scientists are Karl Popper style falsificationists (and in some cases are so unreflectively). The reasoning used in science-as-falsification is similar to the reasoning used in hypothesis testing: assume a theory T to be true, and run experiments to attempt to falsify it. If evidence is gathered that is in discord with T, then we might reject T. The problem here is, again, that was assumed to be true, and no evidence has been gathered in favor of it. Popper thought that, if passed enough severe testing, then was “corroborated”. My hypothesis is that many scientists think that being corroborated actually means that has gained inductive support. That is, scientists think that corroboration of means that is more likely to be true. But Popper did not believe that corroboration meant inductive support. Failing to reject and believing to be (likely) true had too much of a logical leap for Popper. I’m not sure how to get around this leap. But I think that, if we’re being honest, unlike Mnookin and Oliver, we have to admit that the leap exists. That is not to say that mercury in vaccines is linked to any condition (e.g., autism). But, it is to say that, if you are “as positive as one can humanly be that there’s no link” between the two, your confidence extends much further than empirical evidence would allow. That might not be a terrible thing, but it’s something we should acknowledge.


Leave a comment

Filed under Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s