I usually think about this, not as expected utility calculations based on negligible probabilities of vast outcomes being just as likely as their negations, but as them being altogether unreliable, because our numerical intuitions outside the ranges we're calibrated for are unreliable.

For example, when trying to evaluate the plausibility of an extra $500 giving SIAI an extra 1 out of 7 billion chance of succeeding, there is something in my mind that wants to say "well, geez, 1e-10 is such a tiny number, why not?"

Which demonstrates that my brain isn't calibrated to work with numbers in that range, which is no surprise.

So I do best to set aside my unreliable numerical intuitions and look for other tools with which to evaluate that claim.