I'd say it's an error to give weight to any particular highly-improbable scenario without any evidence to distinguish it from the other highly-improbable scenarios. Here's why.
There is a nonzero possibility that some entity will acquire (or already have) godlike powers later today (as per your "I am a god" definition), and decide to use them to increase utility exponentially in response to a number derived somehow from an arbitrary combination of actions by any arbitrary combination of people in the past and the ever-moving present (and let's remember that the requirement could equally well be "condition y is met" or "condition y is not met"). I can't figure out a way to make the number of permutations actually infinite, but considering the negative as well as the positive options makes them cancel out anyway - we have no reason to believe that my posting this comment is more or less likely to trigger a utility increase than my (hypothetically) not posting this comment. The theoretically-possible outcome (huge increase in utility) does not depend on our actions in any predictable way, so there is no reason to modify our actions on this basis.
This leads to the following rational 'conclusion' (specifically considering this issue only) about taking any particular action, on a scale from -1 (definitely don't do) through 0 (indifferent) to 1 (definitely do):
±1/n, n->infinity
(Edit: Actually, this should just be 0. I should lay off the maths when I'm tired.)
(where n is the number of different possible sequences of actions which could possibly trigger the utility increase, and n therefore is unthinkably huge and continues to grow exponentially with each passing second)
Alternatively:
There is also a nonzero possibility etc etc decrease utility etc. etc. Every scenario which could lead to massive increase in utility could instead lead to massive decrease in utility, and we have no way to determine which is less likely.
Tim, you want "a good reason not to be jerked around by unlikely gods in general". Personally I much prefer my first answer (and I suspect you will too), but my alternative answer offers a much more concise rebuttal for any claim of infinite utility increase from an unlikely god:
"Your unlikely god will grant arbitrarily large increase in utility if I take the specified action? Well, my unlikely god will wreak arbitrarily large decrease in utility if I take the specified action. Give me evidence that makes your god and its claim of positive utility more likely than my god and its claim of negative utility, and we can talk - until then the probabilities exactly balance out, so for now I'll just carry on regardless.
This post describes an infinite gamble that, under some reasonable assumptions, will motivate people who act to maximize an unbounded utility function to send me all their money. In other words, if you understand this post and it doesn't motivate you to send me all your money, then you have a bounded utility function, or perhaps even upon reflection you are not choosing your actions to maximize expected utility, or perhaps you found a flaw in this post.
Briefly, we do this with The St. Petersburg Paradox, converted to a mugging along the lines of Pascal's Mugging. I then tweaked it to extract all of the money instead of just a fixed sum.
I have always wondered if any actual payments have resulted from Pascal's Mugging, so I intend to track payments received for this variation. If anyone does have unbounded utility and wants to prove me wrong by sending money, send it with Paypal to tim at fungible dot com. Annotate the transfer with the phrase "St. Petersburg Mugging", and I'll edit this article periodically to say how much money I received. In order to avoid confusing the experiment, and to exercise my spite, I promise I will not spend the money on anything you will find especially valuable. SIAI would be better charity, if you want to do charity, but don't send that money to me.
Here's the hypothetical (that is, false) offer to persons with unbounded utility:
If I am lying and the offer is real, and I am a god, what utility will you receive from sending me a dollar? Well, the probability of me seeing N Tails followed by a Head is (1/2)**(N + 1), and your utility for the resulting universe is UTILITY(UN(N)) >= DUT * 2**N, so your expected utility if I see N tails is (1/2)**(N + 1) * UTILITY(UN(N)) >= (1/2)**(N + 1) * DUT * 2 ** N = DUT/2. There are infinitely many possible values for N, so your total expected utility is positive infinity * DUT/2, which is positive infinity.
I hope we agree that it is unlikely that I am a god, but it's consistent with what you have observed so far, so unless you were born with certain knowledge that I am not a god, you have to assign positive probability to it. Similarly, the probability that I'm lying and the above offer is real is also positive. The product of two positive numbers is positive. Combining this with the result from the previous paragraph, your expected utility from sending me a dollar is infinitely positive.
If you send me one dollar, there will probably be no result. Perhaps I am a god, and the above offer is real, but I didn't do anything beyond flipping the first coin because it came out Tails. In that case, nothing happens. Your expected utility for the next dollar is also infinitely positive, so you should send the next dollar too. By induction you should send me all your dollars.
If you don't send money because you have bounded utility, that's my desired outcome. If you do feel motivated to send me money, well, I suppose I lost the argument. Remember to send all of it, and remember that you can always send me more later.
As of 7 June 2011, nobody has sent me any money for this.
ETA: Some interesting issues keep coming up. I'll put them here to decrease the redundancy: