eli_sennesh comments on Value learning: ultra-sophisticated Cake or Death - Less Wrong

9 Post author: Stuart_Armstrong 17 June 2014 04:36PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (15)

You are viewing a single comment's thread. Show more comments above.

Comment author: [deleted] 19 June 2014 07:30:06PM 0 points [-]

Key reason: value loading AIs do not follow a utility function, but a dynamic construct, that doesn't have all the same properties.

At least as I read the original value-learning paper, they do follow a utility function: the maximum likelihood utility function in some distribution that is subject to Bayesian updating. The hard part was how to construct that distribution and subject it to evidence; the concept that the AI is going to want to have incorrect beliefs (since, after all, the process by which the updates are performed is epistemic, not moral) hadn't occurred to me.