# private_messaging comments on Rationality and Winning - Less Wrong

17 04 May 2012 06:31PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Sort By: Best

Comment author: 07 May 2012 07:01:30AM *  1 point [-]

Adding Y we get by standard updating

P(X | being told Y) = P(being told Y | X) P(X) / P(being told Y).

Even if Y itself is a very strong evidence of X, I needn't necessarily believe Y if I am told Y.

That update is not the problematic one. The problematic is the one where when you are told Y, you add Y itself with some probability set by

P(Y | being told Y) = P(being told Y | Y) P(Y) / P(being told Y).

Then you suddenly have Y in your system (not just 'been told Y') . If you don't do that you can't learn, if you do that you need a lot of hacks not to get screwed over. edit: Or better yet there are hacks that make such agent screw over other agents, as the agent self deludes on some form of pascal's mugging, and tries to broadcast the statement that subverted it, but has hacks not to act in self damaging ways out of such beliefs. For example an agent could invent gods that need to be pleased (or urgent catastrophic problems that needs to be solved), then set up a sacrifice scheme and earn some profits.

Pascal's mugging is a problem for unbounded Bayesian agents as well, it doesn't rely on computation resource limits.

Until unbounded Bayesian agent tells me it got pascal's mugged, that's not really known. I wonder how the Bayesian agent would get the meaning out of pixel values, all way to seeing letters, all way to seeing a message, and then to paying up. Without the 'add hypothesis where none existed before' thing. The unbounded agent got to have pre-existing hypotheses that giving a stranger money will save various numbers of people.

Comment author: 07 May 2012 06:34:35PM 1 point [-]

Then you suddenly have Y in your system (not just 'been told Y') . If you don't do that you can't learn, if you do that you need a lot of hacks not to get screwed over.

I don't think I can't learn if I don't include every hypothesis I am told into my set of hypotheses with assigned probability. A bounded agent may well do some rounding on probabilities and ignore every hypothesis with probability below some threshold.

But even if I include Y with some probability, what does it imply?

Until unbounded Bayesian agent tells me it got pascal's mugged, that's not really known.

Has a bounded agent told you that it got Pacal-mugged? The problem is a combination of a complexity-based prior together with unbounded utility function, and that isn't specific to bounded agents.

Can you show how a Bayesian agent with bounded utility function can be exploited?

Comment author: 08 May 2012 06:17:01AM *  0 points [-]

You're going on the road of actually introducing necessary hacks. That's good. I don't think simply setting threshold probability or capping the utility on a Bayesian agent results in the most effective agent given specific computing time, and it feels to me that you're wrongfully putting a burden of both the definition of what your agent is, and the proof, on me.

You got to define what the best threshold is, or what is the reasonable cap, first - those have to be somehow determined before you have your rational agent that works well. Clearly I can't show that it is exploitable for any values, because assuming hypothesis probability threshold of 1-epsilon and utility cap of epsilon, the agent can not be talked into doing anything at all. edit: and trivially, by setting threshold too low and cap too high, the agent can be exploited.

We were talking about LW rationality. If LW rationality didn't give you procedure for determining the threshold and the cap, then I already demonstrated the point I was making. I don't see huge discussion here on the optimal cap for utility, and on the optimal threshold, and on best handling of the hypotheses below threshold, and it feels to me that rationalists have thresholds set too low and caps set too high. You can of course have an agent that will decide with commonsense and then set threshold and cap as to match it, but that's rationalization not rationality.