Summary:
Pretty reasonable for someone who says "rapture of the nerds". The main problem is anthropomorphism; Stross should read up on optimization processes. There's no reason AIs have to care about themselves to value becoming smarter.
(I've never found a good argument for "AGI is unlikely in theory". It makes me sad, because Stross is looking at practical aspects of uploading, and I need more arguments for/against "AGI is unlikely in practice".)
In some sense, AIs will need to care about themselves-- otherwise they won't adequately keep from damaging themselves as they try to improve themselves, and they won't take measures to protect themselves from outside threats.
The alternative is that they care about their assigned goals, but unless there's some other agent which can achieve their goals better than they can, I don't see a practical difference between AIs taking care of themselves for the sake of the goal and taking care of themselves because that's an independent motivation.
I periodically get email from folks who, having read "Accelerando", assume I am some kind of fire-breathing extropian zealot who believes in the imminence of the singularity, the uploading of the libertarians, and the rapture of the nerds. I find this mildly distressing, and so I think it's time to set the record straight and say what I really think.
Short version: Santa Claus doesn't exist.
- Charles Stross, Three arguments against the singularity, 2011-06-22
EDITED TO ADD: don't get your hopes up, this is pretty weak stuff.