downvoted because you actually said "I would like to do whichever of these two alternatives leads to more utility."

A) no one or almost no one thinks this way, and advice based on this sort of thinking is useless to almost everyone.

B) The entire point of the original post was that, if you try to do this, then you immediately get completely taken over by consideration of any gods you can imagine. When you say that thinking about unlikely gods is not "worth" the computational resources, you are sidestepping the very issue we are discussing. You have already decided it's not worth thinking about tiny probabilities of huge returns.

I think he actually IS making the argument that you assign a low probability to, but instead of dismissing it I think it's actually extremely important to decide whether to take certain courses based on how practical they are. The entire original purpose of this community is research into AI, and while you can't choose your own utility function, you can choose an AI's. If this problem is practically insoluble, then we should design AIs with only bounded utility functions.