timtyler comments on Practical Advice Backed By Deep Theories - Less Wrong

42 Post author: Eliezer_Yudkowsky 25 April 2009 06:52PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (112)

You are viewing a single comment's thread. Show more comments above.

Comment author: Luke_A_Somers 20 November 2011 10:42:35PM *  3 points [-]

Many tests on the same rat can be statistically significant! Do X, Y changes in the rat. Undo it, Y changes back. Repeat until it's statistically certain connection...

We just have no particular reason to expect that it'll generalize well to others.

This really stands out to me as a physicist because we do things like one rat tests all the time. Well, usually we get a few other 'rats', but we rely heavily on the notion that identically prepared matter is... identical. Biology, of course, doesn't allow that shortcut.

Clinicians sometimes have a cohort of 1 for rare diseases... but of course that's simply the best they can do under the circumstances.

Comment author: timtyler 21 November 2011 12:10:58AM 2 points [-]

Many tests on the same rat can be statistically significant! Do X, Y changes in the rat. Undo it, Y changes back. Repeat until it's statistically certain connection...

True - but it won't be too convincing if self-experimenting on yourself with your own diet. Science is based on confirmations of experiments by other scientists.

Comment author: Luke_A_Somers 21 November 2011 03:25:20PM 2 points [-]

The rat being the salesman is the more serious issue there, yes.