= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on Open Thread, August 2010-- part 2 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (369)
While raking, I think I finally thought of a proof that the before-offer-probability can't be known.
The question is basically 'what fraction of all Turing machines making an offer (which is accepted) will then output a certain result?'
We could rewrite this as 'what is the probability that a random Turing machine will output a certain result?
We could then devise a rewriting of all those Turing machines into Turing machines that halt or not when their offer is accepted (eg. halting might = delivering, not halting = welshing on the deal. This is like Rice's theorem).
Now we are asking 'what fraction of all these Turing machines will halt?'
However, this is asking 'what is Chaitin's constant for this rewritten set of Turing machines?' and that is uncomputable!
Since Turing machine-based agents are a subset of all agents that might try to employ Pascal's Mugging (even if we won't grant that agents must be Turing machines), the probability is at least partially uncomputable. A decision procedure which entails uncomputability is unacceptable, so we reject giving the probability in advance, and so our probability must be contingent on the offer's details (like its payoff).
Thoughts?
It seems to be an argument against possibility of making any decision, and hence not a valid argument about this particular decision. Under the same assumptions, you could in principle formalize any situation in this way. (The problem boils down to uncomputability of universal prior itself.)
Besides, not making the decision is not an option, so you have to fall down to some default decision when you don't know how to choose, but where does this default come from?
I take it as an argument against making perfect decisions. If perfection is uncomputable, then any computable agent is not perfect in some way.
The question is what imperfection do we want our agent to have? This might be the deep justification for choosing to scale probability by utility that I was looking for. Scaling linearly corresponds to being willing to lose a fixed amount to mugging, scaling superlinearly correspond to not willing to lose any genuine offer, scaling sublinearly corresponds to not being willing to ever be fooled. Or something like that. The details need some work.