RichardKennaway comments on Superintelligence 9: The orthogonality of intelligence and goals - Less Wrong

8 Post author: KatjaGrace 11 November 2014 02:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (78)

You are viewing a single comment's thread. Show more comments above.

Comment author: RichardKennaway 11 November 2014 02:20:33PM *  1 point [-]

2) Suppose action is the derivative of the utility function in some sense. Then you can derive a utility function from the actions taken in various circumstances. If the 'curl' of the function was not 0, then it was wasting effort. If it was, then it was acting as if it had a utility function anyway.

How do you propose to discover the utility function of an agent by observing its actions? You will only ever see a tiny proportion of the possible situations it could be in, and in those situations you will not observe any of the actions it could have made but did not.

Comment author: Luke_A_Somers 11 November 2014 05:22:46PM 0 points [-]

Observationally, you can't. But given its source code...