thomblake comments on Non-orthogonality implies uncontrollable superintelligence - Less Wrong

14 Post author: Stuart_Armstrong 30 April 2012 01:53PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (45)
Tags: ai control_problem morality superintelligence

Comments (45)

You are viewing a single comment's thread. Show more comments above.

Comment author: thomblake 30 April 2012 07:38:48PM 3 points [-]

As far as I can tell, it's pretty common for moral realists. More or less, the argument goes:

  • Morality is just what one ought to do, so anyone not suffering from akrasia that is correct about morality will do the moral thing
  • A superintelligence will be better than us at knowing facts about the world, like morality
  • (optional) A superintelligence will be better than us at avoiding akrasia
  • Therefore, a superintelligence will behave more morally than us, and will eventually converge on true morality.
Comment author: JGWeissman 30 April 2012 07:47:00PM 8 points [-]

So, the moral realists believe a superintelligence will converge on true morality. Do they also believe that superintelligence is controllable? I had thought they would believe that superintelligence is uncontrollable, but approve of whatever it uncontrollably does.

Comment author: thomblake 30 April 2012 09:12:01PM 3 points [-]

Ah, I missed that clause. Yes, that.

Powered by Reddit Powered by Reddit