Kindly comments on Outside the Laboratory - Less Wrong

63 Post author: Eliezer_Yudkowsky 21 January 2007 03:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (336)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: faul_sname 19 November 2012 03:31:29AM *  11 points [-]

I thought the exact same thing, and wrote a program to test it. Program is below:

 from random import random
p_success = 0.10
def twelve_trials(p_success = 0.25):
>>>># Runs twelve trials, counts the successes
>>>>success_count = 0
>>>>num_trials = 0
>>>>for i in range(12):
>>>>>>>>if random() < p_success:
>>>>>>>>>>>>success_count += 1
>>>>>>>>num_trials += 1
>>>>return success_count
def trials_until_3(p_success = 0.25):
>>>># Runs trials until it hits three successes, counts the trials
>>>>success_count = 0
>>>>num_trials = 0
>>>>while success_count < 3:
>>>>>>>>if random() < p_success:
>>>>>>>>>>>>success_count += 1
>>>>>>>>num_trials += 1
>>>>return num_trials
for i in range(100):
>>>>num_tests = 10000
>>>>twelve_trials_successes = 0
>>>>for i in range(num_tests):
>>>>>>>># See how often there are at least 3 successes in 12 trials
>>>>>>>>twelve_trials_successes += (twelve_trials(p_success) >= 3)
>>>>
>>>>trials_until_3_successes = 0
>>>>for i in range(num_tests):
>>>>>>>># See how often 3 successes happen in 12 trials or less
>>>>>>>>trials_until_3_successes += (trials_until_3(p_success) <= 12)
>>>>print '{0}\t{1}'.format(twelve_trials_successes, trials_until_3_successes)

Turns out they actually are equivalent. I tested with all manner of probabilities of success. Obviously, if what you're actually doing is running a set number of trials in one case and running trials until you reach significance or give up in the second case, you will come up with different results. However, if you have a set number of trials and a set success threshold set beforehand, it doesn't matter whether or not you run all the trials, or just run until the success threshold (which actually seems fairly obvious in retrospect). Edit: formatting sucks

Comment author: Kindly 31 January 2013 04:23:26PM *  4 points [-]

Actually, it's quite interesting what happens if you run trials until you reach significance. Turns out that if you want a fraction p of all trials you do to end up positive, but each trial only ends up positive with probability q<p, then with some positive probability (a function of p and q) you will have to keep going forever.

(This is a well-known result if p=1/2. Then you can think of the trials as a biased random walk on the number line, in which you go left with probability q<1/2 and right otherwise, and you want to return to the place you started. The probability that you'll ever return to the origin is 2q, which is less than 1.)

Comment author: Mqrius 31 January 2013 04:44:32PM *  0 points [-]

Ah, but that's not what it means to run until significance -- in my interpretation in any case. A significant result would mean that you run until you have either p < 0.005 that your hypothesis is correct, or p < 0.005 that it's incorrect. Doing the experiment in this way would actually validate it for "proof" in conventional Science.

Since he mentions "running until you're bored", his interpretation may be closer to yours though.