PhilGoetz comments on The Cognitive Science of Rationality - Less Wrong

88 Post author: lukeprog 12 September 2011 08:48PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (102)

You are viewing a single comment's thread.

Comment author: PhilGoetz 16 September 2011 07:08:02PM *  1 point [-]

Note that having type 1 processes is not an error. AIs will also need to have type 1 processes. Being perfectly rational is never optimal if time and resources are an issue. Being perfectly rational would make sense only given infinite time and resources and no competition.

Comment author: Louie 23 October 2011 11:50:17AM 3 points [-]

I heard you bring this up in person a few times last weekend too. I wanted to follow up with you because I think I'm starting to understand the reason for your disagreements with others on this matter. It is not the case that you're fighting the good fight and we are all just off in crazy land on this issue. I think instead, you're conflating "rationality" with "deliberation".

Naive "perfect rationality" -- being infinitely deliberate -- is of course a mistake. But that's not what Yudkowsky, Omohundro and other careful thinkers are advocating when they discuss wanting to build an AI that is rational. They mean things like having a deliberate rational process that won't knowably violate the axioms of choice or other basic sanity checks that our kludgey, broken minds don't even try to do on their own.

Also, it's possible to design algorithms that are ideally rational within time and resource constraints (ie, AIXItl, Godel Machines). There isn't a false dichotomy between "quick and dirty" heuristic kludges and infinite rationality. You can use "quick and clean" methods which converge towards rationality as they compute, rather than "type 1 processes" which are unrelated to rational deliberation.

Basically, I completely disagree with your first two sentences and would agree with your second two if you replaced "perfectly rational" with words closer to what you actually mean, like "infinitely deliberate".