wedrifid comments on Outside the Laboratory - Less Wrong

63 Post author: Eliezer_Yudkowsky 21 January 2007 03:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (336)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 28 May 2011 08:46:56PM 11 points [-]

The universe doesn't care about Alice's intentions. The trials give information and that information would have been the same even if the trials were run because a rock fell on Alice's keyboard when she wasn't watching.

Surely you accept that if Alice conducts 100 trials and only gives you the successes, you'll get the wrong result no matter the statistical procedure used

Yes, he does.

so you can't say that biased data collection is irrelevant.

Here is where the mistake starts creeping in. You are setting up "biased data collection" to mean selective reporting. Cherry picking the trials that succeed while discarding trials that do not. But in the case of Alice the evidence is all being considered.

You have to either claim that continuing until 3 successes were achieved is an unbiased process, or retreat from the claim that that procedure for collecting the data does not influence the correct interpretation of the results.

The necessary claim is "continuing until 3 successes are achieved does not produce biased data", which is true.

This is a question that is empirically testable. Run a simulation of agents that try to guess, say, which of a set of weighted dice are in use. Pit your 'care what Alice thinks' agents against the bayesian agent. Let them bet among themselves. See which one ends up with all the money.