I would like to extract the meaning of your thought experiment, but it's difficult because the concepts therein are problematic, or at least I don't think they have quite the effect you imagine.
We will define the number choosing game as follows. You name any single finite number x. You then gain x utility and the game then ends. You can only name a finite number, naming infinity is not allowed.
If I were asked (by whom?) to play this game, in the first place I would only be able to attach some probability less than 1 to the idea that the master of the game is actually capable of granting me arbitrarily astronomical utility, and likely to do so. A tenet of the “rationality” that you are calling into question is that 0 and 1 are not probabilities, so if you postulate absolute certainty in your least convenient possible world, your thought experiment becomes very obscure.
E.g. what about a thought experiment in a world where 2+2=5, and also 2+2=4 as well; I might entertain such a thought experiment, but (absent some brilliant insight which would need to be supplied in addition) I would not attach importance to it, in comparison to thought experiments that take place in a world more comprehensible and similar to our own.
Now when I go ahead and attach a probability less than 1—even if it be an extremely high probability—to the idea that the game works just as described, I would become seriously confused by this game because the definition of a utility function is:
A utility function assigns numerical values ("utilities") to outcomes, in such a way that outcomes with higher utilities are always preferred to outcomes with lower utilities.
yet my utility function would, according to my own (meta-...) reflection, with a separate high probability, differ from the utility function that the game master claims I have.
To resolve the confusion in question, I would have to (or would in other terms) resolve confusions that have been described clearly on LessWrong and are considered to be the point at which the firm ground of 21st century human rationality meets speculation. So yes, our concept of rationality has admitted limits; I don't believe your thought experiment adds a new problematic that isn't implied in the Sequences.
How exactly this result applies to our universe isn't exactly clear, but that's the challenge I'll set for the comments.
Bearing in mind that my criticism of your thought experiment as described stands, I'll add that a short story I once read comes to mind. In the story, a modern human finds himself in a room in which the walls are closing in; in the centre of the room is a model with some balls and cup-shaped holders, and in the corner a skeleton of a man in knight's armour. Before he is trapped and suffers the fate of his predecessor, he successfully rearranges the balls into a model of the solar system, gaining utility because he has demonstrated his intelligence (or the scientific advancement of his species) as the alien game master in question would have wished.
If I were presented with a game of this kind, my first response would be to negotiate with the game master if possible and ask him pertinent questions, based on the type of entity he appears to be. If I found that it were in my interests to name a very large number, depending on context I would choose from the following responses:
I have various memories of contemplating the vastness of existence. Please read the most piquant such memory, which I am sure is still encoded in my brain, and interpret it as a number. (Surely "99999..." is only one convenient way of expressing a number or magnitude)
"The number of greatest magnitude that (I, you, my CEV...) (can, would...) (comprehend, deem most fitting...)"
May I use Google? I would like to say "three to the three..." in Knuth's up-arrow notation, but am worried that I will misspell it and thereby fail according to the nature of your game.
Googleplex
"Now when I go ahead and attach a probability less than 1—even if it be an extremely high probability—to the idea that the game works just as described" - You are trying to apply realistic constraints to a hypothetical situation that is not intended to be realistic nor where there are any claims that the results carry over to the real world (as of yet). Taking down an argument I haven't made doesn't accomplish anything.
The gamesmaster has no desire to engage with any of your questions or your attempts to avoid directly naming a number. He simply tells you to just name a number.
In order to ensure that this post delivers what it promises, I have added the following content warnings:
Content Notes:
Pure Hypothetical Situation: The claim that perfect theoretical rationality doesn't exist is restricted to a purely hypothetical situation. No claim is being made that this applies to the real world. If you are only interested in how things apply to the real world, then you may be disappointed to find out that this is an exercise left to the reader.
Technicality Only Post: This post argues that perfectly theoretical rationality doesn't exist due to a technicality. If you were hoping for this post to deliver more, well, you'll probably be disappointed.
Contentious Definition: This post (roughly) defines perfect rationality as the ability to maximise utility. This is based on Wikipedia, which defines rational agents as an agent that: "always chooses to perform the action with the optimal expected outcome for itself from among all feasible actions".
We will define the number choosing game as follows. You name any single finite number x. You then gain x utility and the game then ends. You can only name a finite number, naming infinity is not allowed.
Clearly, the agent that names x+1 is more rational than the agent that names x (and behaves the same in every other situation). However, there does not exist a completely rational agent, because there does not exist a number that is higher than every other number. Instead, the agent who picks 1 is less rational than the agent who picks 2 who is less rational than the agent who picks 3 and so on until infinity. There exists an infinite series of increasingly rational agents, but no agent who is perfectly rational within this scenario.
Furthermore, this hypothetical doesn't take place in our universe, but in a hypothetical universe where we are all celestial beings with the ability to choose any number however large without any additional time or effort no matter how long it would take a human to say that number. Since this statement doesn't appear to have been clear enough (judging from the comments), we are explicitly considering a theoretical scenario and no claims are being made about how this might or might not carry over to the real world. In other words, I am claiming the the existence of perfect rationality does not follow purely from the laws of logic. If you are going to be difficult and argue that this isn't possible and that even hypothetical beings can only communicate a finite amount of information, we can imagine that there is a device that provides you with utility the longer that you speak and that the utility it provides you is exactly equal to the utility you lose by having to go to the effort to speak, so that overall you are indifferent to the required speaking time.
In the comments, MattG suggested that the issue was that this problem assumed unbounded utility. That's not quite the problem. Instead, we can imagine that you can name any number less than 100, but not 100 itself. Further, as above, saying a long number either doesn't cost you utility or you are compensated for it. Regardless of whether you name 99 or 99.9 or 99.9999999, you are still choosing a suboptimal decision. But if you never stop speaking, you don't receive any utility at all.
I'll admit that in our universe there is a perfectly rational option which balances speaking time against the utility you gain given that we only have a finite lifetime and that you want to try to avoid dying in the middle of speaking the number which would result in no utility gained. However, it is still notable that a perfectly rational being cannot exist within a hypothetical universe. How exactly this result applies to our universe isn't exactly clear, but that's the challenge I'll set for the comments. Are there any realistic scenarios where the lack of existence of perfect rationality has important practical applications?
Furthermore, there isn't an objective line between rational and irrational. You or I might consider someone who chose the number 2 to be stupid. Why not at least go for a million or a billion? But, such a person could have easily gained a billion, billion, billion utility. No matter how high a number they choose, they could have always gained much, much more without any difference in effort.
I'll finish by providing some examples of other games. I'll call the first game the Exploding Exponential Coin Game. We can imagine a game where you can choose to flip a coin any number of times. Initially you have 100 utility. Every time it comes up heads, your utility triples, but if it comes up tails, you lose all your utility. Furthermore, let's assume that this agent isn't going to raise the Pascal's Mugging objection. We can see that the agent's expected utility will increase the more times they flip the coin, but if they commit to flipping it unlimited times, they can't possibly gain any utility. Just as before, they have to pick a finite number of times to flip the coin, but again there is no objective justification for stopping at any particular point.
Another example, I'll call the Unlimited Swap game. At the start, one agent has an item worth 1 utility and another has an item worth 2 utility. At each step, the agent with the item worth 1 utility can choose to accept the situation and end the game or can swap items with the other player. If they choose to swap, then the player who now has the 1 utility item has an opportunity to make the same choice. In this game, waiting forever is actually an option. If your opponents all have finite patience, then this is the best option. However, there is a chance that your opponent has infinite patience too. In this case you'll both miss out on the 1 utility as you will wait forever. I suspect that an agent could do well by having a chance of waiting forever, but also a chance of stopping after a high finite number. Increasing this finite number will always make you do better, but again, there is no maximum waiting time.
(This seems like such an obvious result, I imagine that there's extensive discussion of it within the game theory literature somewhere. If anyone has a good paper that would be appreciated).
Link to part 2: Consequences of the Non-Existence of Rationality