RobbBB comments on Bridge Collapse: Reductionism as Engineering Problem - Less Wrong

44 Post author: RobbBB 18 February 2014 10:03PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (61)

You are viewing a single comment's thread. Show more comments above.

Comment author: RobbBB 19 February 2014 08:34:07PM *  3 points [-]

Adele suggested this above. You can see my and Eliezer's response there. The basic worry is that Cartesians have no way to FOOM, because they're unlikely to form intelligent hypotheses about self-modifications. So a real Cartesian won't be an AGI, or will only barely be an AGI. Our work should go into something more useful than that, since it's possible that in the time it takes us to build a moderately useful Cartesian AI that doesn't immediately destroy itself, we could have invented FAI or proto-FAI.

Non-FAI isn't what we're acutely scared of; UFAI (i.e., superintelligence without human values) is. Failing to build a superintelligence is not the same thing as preventing others from building a dangerous superintelligence. So self-handicapping isn't generically useful, especially when most AI researchers won't handicap themselves in the same way.