JGWeissman comments on Non-orthogonality implies uncontrollable superintelligence - Less Wrong

14 Post author: Stuart_Armstrong 30 April 2012 01:53PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (45)
Tags: ai control_problem morality superintelligence

Comments (45)

You are viewing a single comment's thread.

Comment author: JGWeissman 30 April 2012 04:55:58PM 3 points [-]

if you claim that any superintelligence will inevitably converge to some true code of morality, then you are also claiming that no measures can be taken by its creators to prevent this convergence.

That seems obviously true, but what are your motivations for stating it? I was under the impression that people who make the claim accept the conclusion, think it's a good thing, and want to build an AI smart enough to find the "true universal morality" without worrying about all that Friendliness stuff.

Comment author: Stuart_Armstrong 30 April 2012 06:10:37PM *  0 points [-]

It's useful for hitting certain philosophers with. Canonical examples: moral realists sceptical of the potential power of AI.

Comment author: JGWeissman 30 April 2012 06:18:15PM 3 points [-]

There are philosophers who believe that any superintelligence will inevitably converge to some true code of morality, and that superintelligence is controllable? Who?

Comment author: thomblake 30 April 2012 07:38:48PM 3 points [-]

As far as I can tell, it's pretty common for moral realists. More or less, the argument goes:

  • Morality is just what one ought to do, so anyone not suffering from akrasia that is correct about morality will do the moral thing
  • A superintelligence will be better than us at knowing facts about the world, like morality
  • (optional) A superintelligence will be better than us at avoiding akrasia
  • Therefore, a superintelligence will behave more morally than us, and will eventually converge on true morality.
Comment author: JGWeissman 30 April 2012 07:47:00PM 8 points [-]

So, the moral realists believe a superintelligence will converge on true morality. Do they also believe that superintelligence is controllable? I had thought they would believe that superintelligence is uncontrollable, but approve of whatever it uncontrollably does.

Comment author: thomblake 30 April 2012 09:12:01PM 3 points [-]

Ah, I missed that clause. Yes, that.

Comment author: Stuart_Armstrong 30 April 2012 06:33:29PM 2 points [-]

Quite a few I know (not naming names, sorry!) who haven't thought through the implications. Hell, I've only put the two facts together recently in this form.

Powered by Reddit Powered by Reddit