Answers to these questions should be expressed numerically, where possible, but no number should be given without a justification for the specific value.
1. Suppose that you have mislaid your house keys, something most people have experienced at one time or another. You look in various places for them: where you remember having them last, places you've been recently, places they should be, places they shouldn't be, places they couldn't be, places you've looked already, and so on. Eventually, you find them and stop looking.
Every time you looked somewhere, you were testing a hypothesis about their location. You may have looked in a hundred places before finding them.
As a piece of scientific research to answer the question "where are my keys?", this procedure has massive methodological flaws. You tested a hundred hypotheses before finding one that the data supported, ignoring every failed hypothesis. You really wanted each of these hypotheses in turn to be true, and made no attempt to avoid bias. You stopped collecting data the moment a hypothesis was confirmed. When you were running out of ideas to test, you frantically thought up some more. You repeated some failed experiments in the hope of getting a different result. Multiple hypotheses, file drawer effect, motivated cognition, motivated stopping, researcher degrees of freedom, remining of old data: there is hardly a methodological sin you have not committed.
(a) Should these considerations modify your confidence or anyone else's that you have in fact found your keys? If not, why not, and if so, what correction is required?
(b) Should these considerations affect your subsequent decisions (e.g. to go out, locking the door behind you)?
2. You have a lottery ticket. (Of course, you are far too sensible to ever buy such a thing, but nevertheless suppose that you have one. Maybe it was an unexpected free gift with your groceries.) The lottery is to be drawn later that day, the results available from a web site whose brief URL is printed on the ticket. You calculate a chance of about 1 in 100 million of a prize worth getting excited about.
(a) Once the lottery results are out, do you check your ticket? Why, or why not?
(b) Suppose that you do, and it appears that you have won a very large sum of money. But you remember that the prior chance of this happening was 1 in 100 million. How confident are you at this point that you have won? What alternative hypotheses are also raised to your attention by the experience of observing the coincidence of the numbers on your ticket and the numbers on the lottery web site?
(c) Suppose that you go through the steps of contacting the lottery organisers to make a claim, having them verify the ticket, collecting the prize, seeing your own bank confirm the deposit, and using the money in whatever way you think best. At what point, if any, do you become confident that you really did win the lottery? If never, what alternative hypotheses are you still seriously entertaining, to the extent of acting differently on account of them?
As several people have asked about my intentions in posing these problems, I'll answer here.
What I was interested in was seeing how people deal with extreme probabilities.
Some people have in the past expressed the view on LW that it is not humanly possible to be justifiably 80 decibans sure of anything. You would have to able to be right about it with an error rate of no more than 1 in 100 million. Who can be right that often about anything? Surely, some would say, it must remain more likely that you're dreaming, or hypnotised, or being trolled by the Matrix Lords, or something else that you haven't even thought of, for who can scour out every last hundred millionth of possibility space? And yet, ordinary people, who have never learned to believe that it is impossible, have no difficulty in collecting the Euromillions jackpot, which has approximately those odds against. If they are as sure afterwards that they have won as they would have been sure before that they would not, that's a swing of 160 decibans.
BTW, that was the lottery I had in mind in composing the example, and is not a fly-by-night operation. I might have sharpened the example by adding that. Someone wins the Euromillions jackpot every few weeks, for a prize of 10 to above 100 million pounds, depending on how many weeks it has rolled over.
The current consensus in the comments, though, is that the evidence of the house keys is strong enough that the posterior certainty that I have them is not perceptibly swayed by methodological flaws gross enough to completely discredit any paper that relied on statistical techniques to support its claims, and that I can be justifiably sure I have won the lottery at least by the time my bank confirms receipt of the money. These are my own views too.
"0 and 1 are not probabilities", people still say here from time to time, yet a lot of everyday life runs well enough on 0s and 1s.
I think a lot of probabilistic and behavioral reasoning starts to break down and act strangely in the presence of very large odds ratios.
For example, if I discover that I have won the lottery, how should I estimate the probability that I am hallucinating, or dreaming, or insane? In the first case, I cannot trust the evidence of my senses, but I can still reason about that evidence, so I should at least be able to work out a P(hallucination). In the second case, my memory and reasoning faculties are probably significantly impaired, BUT any actions I take wi... (read more)