Are Kolmogorov's difficulty and Hanson's leverage penalty combined into one? Has the more important general problem of probabilities not converging been solved? (The lesswrong posts I looked at did not mention if there is a better solution than the "patch" of the time. Reading of concept also don't give more understanding)
The issue is Adversarial reasoning is basically impossible for Bayesians, so weird results crop up.
This also shows a failure of an assumption in expected value reasoning, that you are logically omniscient and have infinite time to reason about something. This decidedly does not exist in our world, because if we accept logical omniscience, then we can brute force everything by making infinitely complex simulations, thus becoming omnipotent. Pretty obviously, this isn't true, so Pascal's mugging results when we take the approximation as literal truth.
It has been solved in many ways, with different people viewing different solutions as having varying degrees of acceptability and relevance. I don't personally know of anyone who views it as "not acceptably solved", but I'd hesitate to say that it's completely solved in the sense of having one single argument that everyone accepts over all of the others.
Not solved that I know. I personally have solved it by biting the bullet that some branches of the universe are going to be suboptimal, and I'm going to put most of my effort into more likely cases. It doesn't have to converge for me to give up on it.
I think this generalizes - EVERY real agent will be finite, and will violate Bayesean tenets in some cases, like "no 0 or 1 in your probabilities". At some point of unlikelihood, the probability is treated as 0, and there is no evidence that will convince the agent otherwise.
One prong is that I think people have shifted more towards bounded utility functions (infinite ethics arguments are another reason).
It's also plausible that infrabayesianism helps operate in adversarial environments, but maybe not in the obvious way, since you have to set things up so that worst-case reasoning over certain variables gives the right answer.
The situation described in Pascal's mugging is OOD (out-of-distribution) for human values. Human values have not been trained/tested on scenarios with tiny probabilites of vast utilities.
What answer does a system that goes OOD give us? It doesn't matter, we are not supposed to use a system in OOD context.
Naively extrapolating human values too far is not permitted.
Giving an arbitrary/random answer is not permitted.
But we need to make some sort of decision, and we nothing but our values to guide us.
But out values are not defined for the decision we are trying to make.
And we are not allowed to define our values arbitrarily.
I think the answer is really complex, and involves something like "taking all our values and meta-values in account, what is the least arbitrary way we can extend our value system into the space in which we are trying to make a decision"
So, my answer to Pascal's mugging is: human values are probably not yet ready to answer questions like that, at least not in a consistent manner.
Pascal's Mugging isn't OOD. It's very much in-distribution for human beings historically - there is always a scammer waiting on a street corner offering a product that gives you extremely high utility, at very low probability, of course (imagine a tonic that claims to cure smallpox).
Imagine that I, a Lesswrong forum user, claimed to be from outside the simulation and capable of offering you infinite utility (I'm from a universe where that's possible) in exchange for a rare Pepe. That's not a hypothetical offer in a thought experiment. I just did. You...
My understanding of Pascal Mugging is following:
Robber approaches you promising you lots of utility in exchange of giving him $1. The probability he is not lying is extremely low, yet the utility is extremely high, so you give him $1.
The above reasoning has one trivial flaw. How do you know that there isn't a person testing your virtues, which would actually give you lots of utility if you refused to give this person $1? What makes you think that receiving lots of utility when you succumb to the robber is more probable than receiving lots of utility if you stand up to the robber?
The situation described in Pascal's mugging is OOD (out-of-distribution) for human values. Human values have not been trained/tested on scenarios with tiny probabilites of vast utilities.
What answer does a system that goes OOD give us? It doesn't matter, we are not supposed to use a system in OOD context.
Naively extrapolating human values too far is not permitted.
Giving an arbitrary/random answer is not permitted.
But we need to make some sort of decision, and we nothing but our values to guide us.
But out values are not defined for the decision we are trying to make.
And we are not allowed to define our values arbitrarily.
I think the answer is really complex, and involves something like "taking all our values and meta-values in account, what is the least arbitrary way we can extend our value system into the space in which we are trying to make a decision"
So, my answer to Pascal's mugging is: human values are probably not yet ready to answer questions like that, at least not in a consistent manner.
Hmm. You are absolutely right, I didn't think of all these examples.
Let me rephrase:
I think probabilitites on the order of 1/(3^^^3) are OOD for expected utility calculations.
We mostly don't care about expected utility for probabilities that small.
Pascal's mugging is bucketed either into "this is a scam" or "lottery ticket" by human values. And that is fine, unless this results in a contradiction with some of our other values. But I don't think it does.
Extremely high utility yes, extremely low probability no. Usually the idea is that you... (read more)