wedrifid comments on Outside the Laboratory - Less Wrong

63 Post author: Eliezer_Yudkowsky 21 January 2007 03:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (336)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: TimFreeman 28 May 2011 08:18:00PM -2 points [-]

I follow Jaynes in vehemently denying that the posterior probability following an experiment should depend on "whether Alice decided ahead of time to conduct 12 trials or decided to conduct trials until 3 successes were achieved".

I'm sorry, that seems just wrong. The statistics work if there's an unbiased process that determines which events you observe. If Alice conducts trails until 3 successes were achieved, that's a biased process that's sure to ensure that the data ends with a least one success.

Surely you accept that if Alice conducts 100 trials and only gives you the successes, you'll get the wrong result no matter the statistical procedure used, so you can't say that biased data collection is irrelevant. You have to either claim that continuing until 3 successes were achieved is an unbiased process, or retreat from the claim that that procedure for collecting the data does not influence the correct interpretation of the results.

Comment author: wedrifid 28 May 2011 08:46:56PM 11 points [-]

The universe doesn't care about Alice's intentions. The trials give information and that information would have been the same even if the trials were run because a rock fell on Alice's keyboard when she wasn't watching.

Surely you accept that if Alice conducts 100 trials and only gives you the successes, you'll get the wrong result no matter the statistical procedure used

Yes, he does.

so you can't say that biased data collection is irrelevant.

Here is where the mistake starts creeping in. You are setting up "biased data collection" to mean selective reporting. Cherry picking the trials that succeed while discarding trials that do not. But in the case of Alice the evidence is all being considered.

You have to either claim that continuing until 3 successes were achieved is an unbiased process, or retreat from the claim that that procedure for collecting the data does not influence the correct interpretation of the results.

The necessary claim is "continuing until 3 successes are achieved does not produce biased data", which is true.

This is a question that is empirically testable. Run a simulation of agents that try to guess, say, which of a set of weighted dice are in use. Pit your 'care what Alice thinks' agents against the bayesian agent. Let them bet among themselves. See which one ends up with all the money.