Extraordinary claims require extraordinary evidence – rationalwiki gas engine tom


While the idea that a sufficiently outlandish claim requires a lot more compelling evidence is quite intuitive, it can be quantified nicely with probability theory in a Bayesian framework. In short, sufficient evidence must be capable of raising a highly improbable claim to be highly probable – and the more improbable the evidence, the better. By application of Bayes’ theorem, it’s possible to show this in action mathematically.

Assume, for instance, someone claims to be able to predict what way a coin [4] will land almost perfectly. We know this is an extraordinary claim, so we’ll say that just by guessing if the person is telling the truth or not that it’s a million-to-one chance. In reality, the number would be even more improbable, but this can be used for illustration. So we ask them to demonstrate the skill. They’re almost perfect, so let’s assume they guess right about 90% of the time – this allows them the opportunity for their skill to mess up once in a while, but still prove to be pretty good. This gives us all the information we need to know to actually quantify how extraordinary the evidence must be.

A single coin toss doesn’t improve our odds very dramatically. The evidence just isn’t extraordinary enough – you can correctly guess a single coin toss correctly 50% of the time with no special skills involved. It all rests on how improbable our evidence, P(B), actually is and a 50:50 chance isn’t particularly improbable. For two coin tosses P(B) becomes 0.25, and for 10 coin tosses it comes to roughly 0.00097. Plugging those numbers in Bayes’ theorem gives us a probability of genuine skill (given P(A) of a million-to-one) of around 0.0009, which although still small is a considerable improvement on that original million-to-one chance. By 20 or so correctly guessed coin tosses, the skill is starting to look a lot more genuine.

This is the basic idea underpinning statistical significance; is it more likely that our evidence is random, or due to a real effect, and is the improbability of the evidence presented in proportion to the improbability of the claim being made. But Sagan’s quip about extraordinary evidence doesn’t just mean that we can take someone’s word for it if they managed to toss so many coins in a row. Derren Brown can pull off such a feat with some effort and misdirection as shown in his special on The System, so we always need to consider alternative hypotheses and compare how likely they are. Like with Derren Brown tossing a coin with 10 heads in a row, is it more likely that they’re psychic, or are cheating? So tests such as James Randi’s million-dollar challenge will control for this potential factor, making sure that the probability of foul play, fraud and cheating is far less than the probability of genuine psychic power.

The converse of the proposition that ‘extraordinary claims require extraordinary evidence‘ would be that any claim requires some evidence. Therefore, to argue that a competing ordinary claim is more likely to be true than an extraordinary one, simply because the extraordinary one has no "extraordinary" evidence to support it, fails to account for the possibility that the competing "ordinary" claim has no evidence at all to support it (ordinary or otherwise).