Comment on Take heed, for it is a trap

Tue, 16 Aug 2011 22:51:23 GMT

If you're thinking truly reductionistically about programming an AI, you'll realize that "probability" is nothing more than a numerical measure of the amount of information the AI has.

And here I thought it was a numerical measure of how credible it is that the universe looks a particular way. "Probability" is what I plug into expected utility calculations. I didn't realize that I ought to be weighing futures based on "the amount of information" I have about them, rather than how likely they are to come to pass.

Discuss