paulfchristiano comments on Clarification of AI Reflection Problem - Less Wrong

19 Post author: paulfchristiano 10 December 2011 10:30PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (8)

Comments (8)

You are viewing a single comment's thread. Show more comments above.

Comment author: paulfchristiano 11 December 2011 05:42:21PM 1 point [-]
  • I mean (following Shmidhuber) that A' is better than A. Of course if A transforms into A' then A' and A must be about equally good, but what we should really ask is "how much better is A then some less reflective approach?" Then we could imagine large improvements: for example, A could use the first fourth of its time doubling the speed of its proof searcher, the next eighth doubling it again, the next sixteenth doubling it again, and end up finding a better object level solver that would have taken much longer than the entire allotted time to discover using the original proof searcher. The argument I gave suggests that this can't happen in any obvious way, so that A is actually only a factor of 2 faster than a particular non-reflective proof searcher, but that seems to be a different issue.
  • When I wrote "these self-modifications" I meant "these (self-modifications + justifications)", as safety was a property of a justified self-modification. I'll try and clarify.
  • If A can prove that AO always does better with more time, then A can modify into A' and simultaneously commit to spending less time doing proof search. If it spends much more time doing proof search, it runs into the sort of issue I highlighted (consider the additional justified self-modifications it looks at; it can't prove they are all safe, so it can't prove that considering them is good). I could try and make this argument more clear/explicit.
Comment author: Anatoly_Vorobey 11 December 2011 11:47:26PM 0 points [-]

Thanks, this whole part of your post is clearer to me now. I think the post would benefit from integrating these explanations to some degree into the text.

Powered by Reddit Powered by Reddit