Wei_Dai comments on Formalizing Value Extrapolation - Less Wrong

14 Post author: paulfchristiano 26 April 2012 12:51AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (48)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 26 April 2012 11:15:15PM *  3 points [-]

Ultimately we do want the human to be able to run arbitrarily expensive subroutines, which prohibits using any heuristic of the form "stop this computation if it goes on for more than N steps."

What if we keep this heuristic but also define T to have an instruction that is equivalent to calling a halting-problem oracle (with each call counting as one step)? Of course that makes it harder for the outer AGI to reason about how to maximize its utility, but the increase in difficulty doesn't seem very large relative to the difficulty in the original proposal.