= 27d567bc2d48d9f40e9ccc20ea370b15)
Houshalter comments on Pascal's Muggle: Infinitesimal Priors and Strong Evidence - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (404)
Is it just me, or is everyone here overly concerned with coming up with patches for this specific case and not the more general problem? If utilities can grow vastly larger than the prior probability of the situation that contains them, then an expected utility system will become almost useless. Acting on situations with probabilities as tiny as can possibly be represented in that system, since the math would vastly outweigh the expected utility from acting on anything else.
I've heard people come up with apparent resolutions to this problem. Like counter balancing every possible situation with an equally low probability situation that has vast negative utility. There are a lot of problems with this though. What if the utilities don't exactly counterbalance? An extra bit to represent a negative utility for example, might add to the complexity and therefore the prior probability. Or even a tiny amount of evidence for one scenario over the other would completely upset it.
And even if that isn't the case, your utility might not have negative. Maybe you only value the number of paperclips in the universe. The worst that can happen is you end up in a universe with no paperclips. You can't have negative paperclips, so the lowest utility you can have is 0. Or maybe your positive and negative values don't exactly match up. Fear is a better motivator than reward, for example. The fear of having people suffer may have more negative utility than the opposite scenario of just as many people living happy lives or something (and since they are both different scenarios with more differences than a single number, they would have different prior probabilities to begin with.)
Resolutions that involve tweaking the probability of different events is just cheating since the probability shouldn't change if the universe hasn't. It's how you act on those probabilities that we should be concerned about. And changing the utility function is pretty much cheating too. You can make all sorts of arbitrary tweaks that would solve the problem, like having a maximum utility or something. But if you really found out you lived in a universe where 3^^^3 lives existed (perhaps aliens have been breeding extensively, or we really do live in a simulation, etc), are you just supposed to stop caring about all life since it exceeds your maximum amount of caring?
I apologize if I'm only reiterating arguments that have already been gone over. But it's concerning to me that people are focusing on extremely sketchy patches to a specific case of this problem, and not the more general problem, that any expected utility function becomes apparently worthless in a probabilistic universe like ours.
EDIT: I think I might have a solution to the problem and posted it here.
The idea is that it'd be great to have a formalism where they do by construction.
Also, when there's no third party, it's not distinct enough from Pascal's Wager as to demand extra terminology that focusses on the third party, such as "Pascal's Mugging". If it is just agent doing contemplations by itself, that's the agent making a wager on it's hypotheses, not getting mugged by someone.
I'll just go ahead and use "Pascal Scam" to describe a situation where an in-distinguished agent promises unusually huge pay off, and the mark erroneously gives in due to some combination of bad priors and bad utility evaluation. The common errors seem to be 1: omit the consequence of keeping the money for a more distinguished agent, 2: assign too high prior, 3: and, when picking between approaches, ignore the huge cost of acting in a manner which encourages disinformation. All those errors act in favour of the scammer (and some are optional), while non-erroneous processing would assign huge negative utility to paying up even given high priors.
There is no real way of doing that without changing your probability function or your utility function. However you can't change those. The real problem is with the expected utility function and I don't see any way of fixing it, though perhaps I missed something.
Any agent subject to Pascal's Mugging would fall pray to this problem first, and it would be far worse. While the mugger is giving his scenario, the agent could imagine an even more unlikely scenario. Say one where the mugger actually gives him 3^^^^^^3 units of utility if he does some arbitrary task, instead of 3^^^3. This possibility immediately gets so much utility that it far outweighs anything the mugger has to say after that. Then the agent may imagine an even more unlikely scenario where it gets 3^^^^^^^^^^3 units of utility, and so on.
I don't really know what an agent would do if the expected utility of any action approached infinity. Perhaps it would generally work out as some things would approach infinity faster than others. I admit I didn't consider that. But I don't know if that would necessarily be the case. Even if it is it seems "wrong" for expected utilities of everything to be infinite and only tiny probabilities to matter for anything. And if so then it would work out for the pascal's mugging scenario too I think.
Last time I checked, priors were fairly subjective even here. We don't know what is the best way to assign priors. Things like "Solomonoff induction" depend to arbitrary choice of machine.
Nope, people who end up 419-scammed or waste a lot of money investing into someone like Randel L Mills or Andrea Rossi live through their life ok until they read a harmful string in a harmful set of circumstances (bunch of other believers around for example).
Priors are indeed up for grabs, but a set of priors about the universe ought be consistent with itself, no? A set of priors based only on complexity may indeed not be the best set of priors -- that's what all the discussions about "leverage penalties" and the like are about, enhancing Solomonoff induction with something extra. But what you seem to suggest is a set of priors about the universe that are designed for the express purposes of making human utility calculations balance out? Wouldn't such a set of priors require the anthroporphization of the universe, and effectively mean sacrificing all sense of epistemic rationality?
The best "priors" about the universe are 1 for what that universe right around you is, and 0 for everything else. Other priors are a compromise, an engineering decision.
What I am thinking is that
there is a considerably better way to assign priors which we do not know of yet - the way which will assign equal probabilities to each side of a die if it has no reason to prefer one over the other - the way that does correspond to symmetries in the evidence.
We don't know that there will still be same problem when we have a non-stupid way to assign priors (especially as the non-stupid way ought to be considerably more symmetric). And it may be that some value systems are intrinsically incoherent. Suppose you wanted to maximize blerg without knowing what blerg even really is. That wouldn't be possible, you can't maximize something without having a measure of it. But I still can tell you i'd give you 3^^^^3 blergs for a dollar, without either of us knowing what blerg is supposed to be or whenever 3^^^^3 blergs even make sense (if blerg is an unique good book of up to 1000 page length, it doesn't because duplicates aren't blerg).
True, but the goal of a probability function is to represent the actual probability of an event happening as closely as possible. The map should correspond to the territory. If your map is good, you shouldn't change it unless you observe actual changes in the territory.
I don't know if those things have such extremes in low probability vs high utility to be called pascal's mugging. But even so, the human brain doesn't operate on anything like Solomonoff induction, Bayesian probability theory, or expected utility maximization.
The actual probability is either 0 or 1 (either happens or doesn't happen). Values in-between quantify ignorance and partial knowledge (e.g. when you have no reason to prefer one side of the die to the other), or, at times, are chosen very arbitrarily (what is the probability that a physics theory is "correct").
New names for same things are kind of annoying, to be honest, especially ill chosen... if it happens by your own contemplation, I'd call it Pascal's Wager. Mugging implies someone making threats, scam is more general and can involve promises of reward. Either way the key is the high payoff proposition wrecking some havoc, either through it's prior probability being too high, other propositions having been omitted, or the like.
People are still agents, though.
Yes but the goal is to assign whatever outcome that will actually happen with the highest probability as possible, using whatever information we have. The fact that some outcomes result in ridiculously huge utility gains does not imply anything about how likely they are to happen, so there is no reason that should be taken into account (unless it actually does, in which case it should.)
Pascal's mugging was an absurd scenario with absurd rewards that approach infinity. What you are talking about is just normal everyday scams. Most scams do not promise such huge rewards or have such low probabilities (if you didn't know any better it is feasible that someone could have an awesome invention or need your help with transaction fees.)
And the problem with scams is that people overestimate their probability. If they were to consider how many emails in the world are actually from Nigerian Princes vs scammers, or how many people promise awesome inventions without any proof they will actually work, they would reconsider. In pascal's mugging, you fall for it even after having considered the probability of it happening in detail.
Your probability estimation could be absolutely correct. Maybe 1 out of a trillion times a person meets someone claiming to be a matrix lord, they are actually telling the truth. And they still end up getting scammed, so that the 1 in a trillionth counter-factual of themselves gets infinite reward.
They are agents, but they aren't subject to this specific problem because we don't really use expected utility maximization. At best maybe some kind of poor approximation of it. But it is a problem for building AIs or any kind of computer system that makes decisions based on probabilities.
I think you're considering a different problem than Pascal's Mugging, if you're taking it as a given that the probabilities are indeed 1 in a trillion (or for that matter 1 in 10). The original problem doesn't make such an assumption.
What you have in mind, the case of definitely known probabilities, seems to me more like The LifeSpan dilemma where e.g. "an unbounded utility on lifespan implies willingness to trade an 80% probability of living some large number of years for a 1/(3^^^3) probability of living some sufficiently longer lifespan"
The wiki page on it seems to suggest that this is the problem.
Also this
which is pretty concerning.
I'm curious what you think the problem with Pascal's Mugging is though. That you can't easily estimate the probability of such a situation? Well that is true of anything and isn't really unique to Pascal's Mugging. But we can still approximate probabilities. A necessary evil to live in a probabilistic world without the ability to do perfect Bayesian updates on all available information, or unbiased priors.
I abhor using unnecessary novel jargon.
Bad math being internally bad, that's the problem. Nothing to do with any worlds, real or imaginary, just a case of internally bad math - utilities are undefined, it is undefined if you pay up or not, the actions chosen are undefined. Akin to maximizing blerg without any definition of what blerg even is - maximizing "expected utility" without having defined it.
Speed prior works, for example (it breaks some assumptions of Blanc. Namely, the probability is not bounded from below by any computable function of length of the hypothesis).
There is no evidence for the actual existence of neatly walled-of and unupdateable utility functions or probability functions, any more than there is for a luz'.
Utility and probability functions are not perfect or neatly walled off. But that doesn't mean you should change them to fix a problem with your expected utility function. The goal of a probability function is to represent the actual probability of an event happening as closely as possible. And the goal of a utility function is to represent what you states you would prefer the universe to be in. This also shouldn't change unless you've actually changed your preferences.
There's plenty of evidence of people changing their preferences over significant periods of time: it would be weird not to. And I am well aware that the theory of stable utility functions is standardly patched up with a further theory of terminal values, for which there is also no direct evidence.
Of course people can change their preferences. But if your preferences are not consistent you will likely end up in situations that are less preferable than if you had the same preferences the entire time. It also makes you a potential money pump.
What? Terminal values are not a patch for utility functions. It's basically another word that means the same thing, what state you would prefer the world to end up in. And how can there be evidence for a decision theory?
Well, I've certainly seen discussions here in which the observed inconsistency among our professed values is treated as a non-problem on the grounds that those are mere instrumental values, and our terminal values are presumed to be more consistent than that.
Insofar as stable utility functions depend on consistent values, it's not unreasonable to describe such discussions as positing consistent terminal values in order to support a belief in stable utility functions.
Well, how is this different from changing our preferences to utility functions to fix problems with our naive preferences?
I don't know what you mean. All I'm saying is that you shouldn't change your preferences because of a problem with your expected utility function. Your preferences are just what you want. Utility functions are just a mathematical way of expressing that.
Human preferences don't naturally satisfy the VNM axioms, thus by expressing them as a utility function you've already changed them.
I don't see why our preferences can't be expressed by a utility function even as they are. The only reason it wouldn't work out is if there were circular preferences, and I don't think most peoples preferences would work out to be truly circular if they were to think about the specific occurrence and decide what they really preferred.
Though mapping out which outcomes are more preferred than others is not enough to assign them an actual utility, you'd somehow have to guess how much more preferable one outcome is to another quantitatively.But even then I think most people could if they thought about it enough. The problem is that our utility functions are complex and we don't really know what they are, not that they don't exist.
Or they might violate the independence axiom, but in any case what do you mean by " think about the specific occurrence and decide what they really preferred", since the result of such thinking is likely to depend on the exact order they thought about things in.