You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

JenniferRM comments on Can anyone explain to me why CDT two-boxes? - Less Wrong Discussion

-12 Post author: Andreas_Giger 02 July 2012 06:06AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (136)

You are viewing a single comment's thread. Show more comments above.

Comment author: JenniferRM 02 July 2012 10:40:26PM *  6 points [-]

You seem to be fighting the hypothetical, but I don't know if you're doing it out of mistrust or because some background would be helpful. I'll assume helpful background would be helpful... :-)

A program could be designed to (1) search for relevant sensory data within a larger context, (2) derive a mixed strategy given the input data, (3) gets more bits of salt from local thermal fluctuations than log2(number of possible actions), (4) drop the salt into a pseudo-random number generator over its derived mixed strategy, and (5) output whatever falls out as its action. This rough algorithm seems strongly deterministic in some ways, and yet also strongly reminiscent of "choice" in others.

This formulation reduces the "magic" of Omega to predicting the relatively fixed elements of the agent (ie, steps 1, 2, and 4) which seems roughly plausible as a matter of psychology and input knowledge and so on, and also either (A) knowing from this that the strategy that will be derived isn't actually mixed so the salt is irrelevant, or else (B) having access/control of the salt in step 3.

In AI design, steps 1 and 2 are under the programmer's control to some degree. Some ways of writing the program might make the AI more or less tractable/benevolent/functional/wise and it seems like it would be good to know which ways are likely to produce better outcomes before any such AI is built and achieves takeoff rather than after. Hence the interest in this thought experiment as an extreme test case. The question is not whether step 3 is pragmatically possible for an imaginary Omega to hack in real life. The question is how to design steps 1 and 2 in toy scenarios where the program's ability to decide how to pre-commit and self-edit are the central task, so that harder scenarios can be attacked as "similar to a simpler solved problem".

If you say "Your only choices are flipping a coin or saying a predetermined answer" you're dodging the real question. You can be dragged back to the question by simply positing "Omega predicts the coin flip, what then?" If there's time and room for lots and lots of words (rather than just seven words) then another way to bring attention back to the question is to explain about fighting the hypothetical, try to build rapport, see if you can learn to play along so that you can help advance a useful intellectual project.

If you still "don't get it", then please, at least don't clog up the channel. If you do get it, please offer better criticism. Like, if you know of a different but better thought experiment where effectively-optimizing self-modifying pre-commitment is the central feature of study, that would be useful.