Once upon a time, there was a court jester who dabbled in logic.
The jester presented the king with two boxes. Upon the first box was inscribed:
"Either this box contains an angry frog, or the box with a false inscription contains an angry frog, but not both."
On the second box was inscribed:
"Either this box contains gold and the box with a false inscription contains an angry frog, or this box contains an angry frog and the box with a true inscription contains gold."
And the jester said to the king: "One box contains an angry frog, the other box gold; and one, and only one, of the inscriptions is true."
The king opened the wrong box, and was savaged by an angry frog.
"You see," the jester said, "let us hypothesize that the first inscription is the true one. Then suppose the first box contains gold. Then the other box would have an angry frog, while the box with a true inscription would contain gold, which would make the second statement true as well. Now hypothesize that the first inscription is false, and that the first box contains gold. Then the second inscription would be—"
The king ordered the jester thrown in the dungeons.
A day later, the jester was brought before the king in chains, and shown two boxes.
"One box contains a key," said the king, "to unlock your chains; and if you find the key you are free. But the other box contains a dagger for your heart, if you fail."
And the first box was inscribed:
"Either both inscriptions are true, or both inscriptions are false."
And the second box was inscribed:
"This box contains the key."
The jester reasoned thusly: "Suppose the first inscription is true. Then the second inscription must also be true. Now suppose the first inscription is false. Then again the second inscription must be true. So the second box must contain the key, if the first inscription is true, and also if the first inscription is false. Therefore, the second box must logically contain the key."
The jester opened the second box, and found a dagger.
"How?!" cried the jester in horror, as he was dragged away. "It's logically impossible!"
"It is entirely possible," replied the king. "I merely wrote those inscriptions on two boxes, and then I put the dagger in the second one."
(Adapted from Raymond Smullyan.)
The simplest way to solve the jester's puzzle is to make a table of the four cases (where the frog is, where the true inscription is), then determine for each case whether the inscriptions are in fact true or false as required for that case. (All the while making la-la-la-can't-hear-you noises at any doubts one might have about whether self-reference can be formalised at all.) The conclusion is that the first box has the frog and the true inscription. That assumes that the jester was honest in stating his puzzle, but given his shock at the outcome of the king's puzzle, that appears to be so.
But can self-reference be formalised? How, for example, do two perfect reasoners negotiate a deal? In general, how can two perfect reasoners in an adversarial situation ever interpret the other's words as anything but noise?
"Are you the sort of man who would put the poison into his own goblet or his enemy's? Now, a clever man would put the poison into his own goblet because he would know that only a great fool would reach for what he was given. I am not a great fool so I can clearly not choose the wine in front of you...But you must have known I was not a great fool; you would have counted on it, so I can clearly not choose the wine in front of me." ...etc.
Or consider a conversation between an FAI that wants to keep the world safe for humans, and a UFAI that wants to turn the world into paperclips.