MrMind comments on Depth-based supercontroller objectives, take 2 - LessWrong

2 Post author: sbenthall 24 September 2014 01:25AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (21)

Comments (21)

You are viewing a single comment's thread.

Comment author: MrMind 23 September 2014 10:15:23AM *  2 points [-]

Culture, thought, human DNA, human values, etc. have been stripped to their functional carbon and hydrogen atoms and everything now just optimizes for paperclip manufacturing or whatever. D(u/r) = D(u)

I contest this derivation. Whatever process produced humanity, made so that humanity produced an unsafe supercontroller. This may means that whatever the supercontroller is optimized for, it's part of the process that produced humanity, and so it does not make g(u,h) go to zero.

Of course, without a concrete model, it's impossible to say for certain.

Comment author: sbenthall 23 September 2014 05:47:13PM 2 points [-]

So, the key issue is whether or not the representations produced by the paperclip optimizer could have been produced by other processes. If there is another process that produces the paperclip-optimized representations more efficiently than going through the process of humanity, then that process dominates the calculation of D(r).

In other words, for this objection to make sense, it's not enough for the humanity to have been sufficient for the R scenario. It must be necessary for producing R, or at least necessary to result in it in the most efficient possible way.

What are your criteria for a more concrete model than what has been provided?

Powered by Reddit Powered by Reddit