Eliezer_Yudkowsky comments on Optimization - Less Wrong

20 Post author: Eliezer_Yudkowsky 13 September 2008 04:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (44)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Eliezer_Yudkowsky 13 September 2008 05:12:51PM 7 points [-]

Robin, the circumstances under which a Bayesian will come to believe that a system is optimizing, are the same circumstances under which a message-length minimizer will send a message describing the system's "preferences": namely, when your beliefs about its preferences are capable of making ante facto predictions - or at least being more surprised by some outcomes than by others.

Most of the things we find it useful to describe as optimizers, have preferences that are stable over a longer timescale than the repeated observations we make of them. An inductive suspicion of such stability is enough of a prior to deal with aliens (or evolution). Also, different things of the same class often have similar preferences, like multiple humans.