It's not just "very large payoff and very small probability". Take a bunch of events of the form "N people get tortured". if you use Solomonoff induction, your prior probability for these events will roughly decrease in log(N). However, if you aggregate human suffering linearly, the utility you assign to these events increases in N (otherwise, find events with utility linear in N - they exist if your utility function is unbounded). Therefore, your expected utility diverges as N goes to infinity. So for any certain cost, there's a number N large enough for you to pay the cost.

But this is not the case for the lottery. The payoff and its probability are known, and the expected gain is much less than the cost.

The problem with Pascal's Mugging is that we think the expected payoff of certain kinds of actions is huge (which probably reveals a problem with how we compute expected utility). The problem with lotteries is that we know the expected payoff is negative, but play anyway.