Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

FrankAdamek comments on Deciding on our rationality focus - Less Wrong

34 22 July 2009 06:27AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (48)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: 22 July 2009 03:47:20PM 0 points [-]

people who are rational-neutral discard evidence regarding statements, as any intelligent being must

I feel like I should be able to find this out on my own, but I've had no success so far. Does "evidence regarding statements" refer to statements that are evidence-regarding, or evidence that regards statements? Either way I can't figure out an obvious reason to reject such things. Is it the idea that evidence shouldn't be discussed on any aspect beyond validity? I feel I'm missing something, many thanks to anyone who can throw me a link or other resource.

Comment author: 22 July 2009 09:55:27PM *  4 points [-]

(If this post is too long, read only the last paragraph.)

Evidence that regards statements. I guess the "regarding statements" bit was redundant. Anyway, let me try to give some examples.

First, let me postulate a guy named Delta. Delta is an extremely rational robot who, given the evidence, always comes up with the best possible conclusion.

Andy the Apathetic is presented with a court case. Before he ever looks at the case, he decides that the probability the defendant is guilty is 50%. In fact, he never looks at the case; he tosses it aside and gives that 50% as his final judgement. Andy is rational-neutral, as he discarded evidence regardless of its direction; his probability is useless, but if I told Delta how Andy works and Andy's final judgement, Delta would agree with it.

Barney the Biased is presented with the same court case. Before he ever looks at the case, he decides that the probability that the defendant is guilty is 50%. Looking through the evidence, he decides to discard everything suggesting that the defendant is innocent; he concludes that the defendant has a 99.99% chance of being guilty and gives that as his final judgement. Barney is not rational-neutral, as he discarded evidence with regard to its direction; his probability is almost useless (but not as useless as Andy's), and if I told Delta how Barney works and Barney's final judgement, Delta might give a probability of only 45%.

Finally, Charlie the Careful is presented with the same court case. Before he ever looks at the case, he decides that the probability that the defendant is guilty is 50%. Looking through the evidence, he takes absolutely everything into account, running the numbers and keeping Bayes' law between his eyes at all times; eventually, after running a complete analysis, he decides that the probability that the defendant is guilty is 23.14159265%. Charlie is rational-neutral, as he discarded evidence regardless of its direction (in fact, he discarded no evidence); if I told Delta how Charliie works and Charlie's final judgement, Delta would agree with it.

So, here's another definition of rational neutrality I came up with by writing this: you are rational-neutral if, given only your source code, it's impossible to come up with a function that takes one of your probability estimates and returns a better probability estimate.

Comment author: 22 July 2009 11:55:12PM 1 point [-]

It might be useful to revise this concept to account for computational resources (see AI work on 'limited rationality', e.g. Russell and Wefald's "Do the Right Thing" book).

Comment author: 23 July 2009 08:58:09PM 0 points [-]

I'll try my best to get my hands on a copy of that book.

Comment author: 23 July 2009 06:08:28AM 0 points [-]

Upon thinking about that second definition of rational neutrality, I find myself thinking that that can't be right. It's identical to calibration. And even a rational-neutral agent that's been "repaired" by applying the best possible probability estimate adjustment function will still return the same ordinal probabilities: Barney the Biased, even after adjustment, will return higher probabilities for statements he is biased toward than statements he is biased against.

I would have said this:

So, here's another definition of rational neutrality I came up with by writing this: you are rational-neutral if, given only your source code and your probability estimates, it's impossible for someone to come up with better probability estimates.

...but that definition doesn't rule out the possibility that an agent would look at your probability estimates, figure out what the problem is, and come up with a better solution on its own. In the extreme case, no agent would be considered rational-neutral unless it had a full knowledge of all mathematical results. That's not what I want; therefore, I stick by my original definition.

Comment author: 22 July 2009 11:30:46PM 0 points [-]

It took two read-throughs to get this, but I'm fairly sure that's the concept and not your handling.Thanks for the explanation!