An update to this post

It appears that this issue has been discussed before in the thread Naturalism versus unbounded (or unmaximisable) utility options. The discussion there didn't end up drawing the conclusion that perfect rationality doesn't exist, so I believe this current thread adds something new.

Instead, the earlier thread considers the Heaven and Hell scenario where you can spend X days in Hell to get the opportunity to spend 2X days in Heaven. Most of the discussion on that thread was related to the limit of how many days an agent count so as to exit at some point. Stuart Armstrong also comes up with the same solution for demonstrating that this problem isn't related to unbounded utility.

Qiaochu Yaun summarises one of the key takeaways: "This isn't a paradox about unbounded utility functions but a paradox about how to do decision theory if you expect to have to make infinitely many decisions. Because of the possible failure of the ability to exchange limits and integrals, the expected utility of a sequence of infinitely many decisions can't in general be computed by summing up the expected utility of each decision separately."

Cudos to Andreas Giger for noticing what most of the commentators seemed to miss: "How can utility be maximised when there is no maximum utility? The answer of course is that it can't." This is incredibly close to stating that perfect rationality doesn't exist, but it wasn't explicitly stated, only implied.

Further, Wei Dai's comment on a randomised strategy that obtains infinite expected utility is an interesting problem that will be addressed in my next post.