wedrifid comments on Knowledge value = knowledge quality × domain importance - Less Wrong

8 Post author: John_Maxwell_IV 16 April 2012 08:40AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (40)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 16 April 2012 03:38:46PM 1 point [-]

(Agreeing and elaborating.)

Methodologically, each self-experiment is typically much more poorly run than the kinds of trials we try to discuss here (RCTs), so each self-experiment represents less than n=1 of data. The RCTs usually have at least a few dozen and ideally hundreds or thousands of subjects, either singly or pooled for meta-analysis. So a single such meta-analysis represents thousands of subjects times the fractional quality of a self-experiment, leading to the conclusion that one self-experiment is worth somewhere less than one-hundredths to one-thousandths of any comparable RCT.

The benefits of self experimentation in areas where there has already been actual bulk research to the level where meta analysis can be done is in the realm of personalizing - finding the effects on yourself in particular. Even then the degree to which self experiment can cause you to update the predicted benefits to yourself will depend on the degree to which variability between people is found (unfortunately not mentioned too much in most studies) and the degree to which self reports (or whatever metrics you use) tend to be reliable indicators of actual influence.

This means that if I am self-experimenting with modafinil I will update significantly on how useful the substance is to myself while I wouldn't even bother self experimenting with respect to how much background noise polution influences my general wellbeing over an extended period. In the latter case p(background sound is detrimental | my self experiment indicates background sound is beneficial) > 0.7.