David_Gerard comments on Knowledge value = knowledge quality × domain importance - Less Wrong

8 Post author: John_Maxwell_IV 16 April 2012 08:40AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (40)

Comments (40)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 16 April 2012 03:15:45PM *  6 points [-]

But knowledge does not come in two grades, "scientific" and "useless". Anecdotes do count as Bayesian evidence, they are just weak Bayesian evidence. And well designed scientific studies constitute stronger Bayesian evidence then poorly designed studies. There's a continuum for knowledge quality.

Methodologically, each self-experiment is typically much more poorly run than the kinds of trials we try to discuss here (RCTs), so each self-experiment represents less than n=1 of data. The RCTs usually have at least a few dozen and ideally hundreds or thousands of subjects, either singly or pooled for meta-analysis. So a single such meta-analysis represents thousands of subjects times the fractional quality of a self-experiment, leading to the conclusion that one self-experiment is worth somewhere less than one-hundredths to one-thousandths of any comparable RCT.

The human mind doesn't do 64-bit floating weights. It doesn't even do shorts.

Comment author: David_Gerard 16 April 2012 04:33:51PM 0 points [-]

The human mind doesn't do 64-bit floating weights. It doesn't even do shorts.

BTW, has anyone ascertained what resolution it does do? (Is this even a coherent question?)

Comment author: gwern 16 April 2012 04:40:32PM *  5 points [-]

Well, it's a question which could be turned into a coherent question in a couple ways, so before getting an answer, you need to decide what question you're asking and what an answer ought to look like. For example:

  • You could ask whether people can distinguish between biased dice down to single percent level or smaller by rolling them a ton of times.
  • You could ask whether calibrated experts can be calibrated down to sub-percent levels without resorting to explicit models and calculation, or whether the inherent mental noise overwhelms differentials before then.
  • You could try to tie it to pulse-coding for utility/rewards (lukeprog covered in one of his neuroscience posts) which would imply something like nothing finer than 1/1000th or something. And so on.

I don't know the answers to any of these - my own impression is that people have fairly granular probabilities. I don't bother with single-percent differences in my own predictions on PredictionBook.com unless I'm in the 0-10/90-100% decile (where 0% is quite different from 1%).

Comment author: TheOtherDave 16 April 2012 05:25:59PM 1 point [-]

Hrm.

Rolling dice a ton of times starts running into problems with short-term memory buffer size and conflation with explicit strategies for managing that limit; it might be more useful to provide a histogram of the results of a hundred die rolls and ask whether it's a biased die or not.

Though, thinking about this... surely this isn't an absolute granularity? I mean, even supposing that it's constant at all. I would expect the minimum size of a detectable probability shift to be proportional to the magnitude of the original probability.

Comment author: David_Gerard 16 April 2012 05:04:35PM *  0 points [-]

This is a question I've thought of posting in discussion before, but I couldn't work out a coherent phrasing. Just how well can the untrained human mind resolve probabilities? Just how well can the trained human mind (e.g. say, a professional bookmaker) resolve probabilities? (Note I have no idea how individual bookmakers do things these days, for all I know they routinely use computers rather than estimating odds themselves. I know the chain ones do.)

Powered by Reddit Powered by Reddit