Your understanding of "strictly bounded" is artificial, and not what I was talking about. I was talking about assigning a strict, numerical bound to utility. That does not prevent having an infinite number of values underneath that bound.
Isn't that the same as an asymptote, which I talked about?
It would be silly to assign a bound and a function low enough that "You wouldn't take a 1% risk of losing 1,000 people, to save 2,000 people otherwise," if you meant this literally, with these values.
You can set the bound wherever you want. It's arbitrary. My point is that if you ever reach it, you start behaving weird. It is not a very natural fix. It creates other issues.
It is true that if you increase the values enough, something like that will happen. And that is exactly the way real people would behave, as well.
Maybe human utility functions are bounded. Maybe they aren't. We don't know for sure. Assuming they are is a big risk. And even if they are bounded, it doesn't mean we should put that into an AI. If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should.
"If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should."
I thought the whole idea behind this proposal was that the probability of this happening is essentially zero.
If you think this is something with a reasonable probability, you should accept the mugging.
Summary: the problem with Pascal's Mugging arguments is that, intuitively, some probabilities are just too small to care about. There might be a principled reason for ignoring some probabilities, namely that they violate an implicit assumption behind expected utility theory. This suggests a possible approach for formally defining a "probability small enough to ignore", though there's still a bit of arbitrariness in it.