= 27d567bc2d48d9f40e9ccc20ea370b15)
MixedNuts comments on Charles Stross: Three arguments against the singularity - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (54)
Summary:
Pretty reasonable for someone who says "rapture of the nerds". The main problem is anthropomorphism; Stross should read up on optimization processes. There's no reason AIs have to care about themselves to value becoming smarter.
(I've never found a good argument for "AGI is unlikely in theory". It makes me sad, because Stross is looking at practical aspects of uploading, and I need more arguments for/against "AGI is unlikely in practice".)
In some sense, AIs will need to care about themselves-- otherwise they won't adequately keep from damaging themselves as they try to improve themselves, and they won't take measures to protect themselves from outside threats.
The alternative is that they care about their assigned goals, but unless there's some other agent which can achieve their goals better than they can, I don't see a practical difference between AIs taking care of themselves for the sake of the goal and taking care of themselves because that's an independent motivation.
Sounds like he doesn't believe in the possibility of nonperson predicates.
No, it seems to be a different mistake. He thinks nonperson AIs are possible, but they will model themselves as... roughly, body parts of humans. So they won't optimize for anything, just obey explicit orders.