It may very well be that a utility maximizer will always be subject to some form of possible mugging.
I fear that the mugger is often our own imagination. If you calculate the expected utility of various outcomes you imagine impossible alternative actions. The alternatives are impossible because you already precommited to choosing the outcome with the largest expected utility. There are three main problems with that:
All this can cause any insignificant inference to exhibit hyperbolic growth in utility.
I don't trust my brain's claims of massive utility enough to let it dominate every second of my life. I don't even think I know what, this second, would be doing the most to help achieve a positive singularity.
I'm also pretty sure that my utility function is bounded, or at least hits diminishing returns really fast.
I know that thinking my head off about every possible high-utility counterfactual will make me sad, depressed, and indecisive, on top of ruining my ability to make progress towards gaining utility.
So I don't worry about it that much. I try to think about these problems in doses that I can handle, and focus on what I can actually do to help out.
So after reading SarahC's latest post I noticed that she's gotten a lot out of rationality.
More importantly, she got different things out of it than I have.
Off the top of my head, I've learned...
Where she got...
I've only recently making a habit out of trying new things, and that's been going really well for me. Is there other low hanging fruit that I'm missing?
What cool/important/useful things has rationality gotten you?