snarles comments on Steelmaning AI risk critiques - Less Wrong

26 Post author: Stuart_Armstrong 23 July 2015 10:01AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (98)

You are viewing a single comment's thread. Show more comments above.

Comment author: jacob_cannell 24 July 2015 04:42:05AM *  9 points [-]

Here is a novel argument you may or may not have heard: We live in the best of all probable worlds due to simulation anthropics. Future FAI civs spend a significant amount of their resources to resimulate and resurrect past humanity - winning the sim race by a landslide (as UFAI is not strongly motivated to sim us in large numbers). As a result of this anthropic selection force, we find ourselves in a universe that is very lucky - it is far more likely to lead to FAI than you would otherwise think.

The best standard argument is this: the brain is a universal learning machine - the same general architecture that will necessarily form the basis for any practical AGI. In addition the brain is already near optimal in terms of what can be done for 10 watts with any irreversible learning machine (this is relatively easy to show from wiring energy analysis). Thus any practical AGI is going to be roughly brain like, similar to baby emulations. All of the techniques used to raise humans safely can thus be used to raise AGI safely. LW/MIRI historically reject this argument based - as far as I can tell - on a handwavey notion of 'anthropomorphic bias', which has no technical foundation.

I've presented the above argument about four years ago, but I never bothered to spend the time backing it up in excruciating formal detail. Until more recently. The last 5 years of progress in AI strongly supports this anthropomorphic AGI viewpoint.

Comment author: snarles 25 July 2015 04:05:59AM *  5 points [-]

There is no way to raise a human safely if that human has the power to exponentially increase their own capabilities and survive independently of society.

Comment author: kokotajlod 26 July 2015 08:35:56PM 0 points [-]

Yep. "The melancholy of haruhi suzumiya" can be thought of as an example of something in the same reference class.