I can't speak for Eliezer's intentions when he wrote this story, but I can see an incredibly simple moral to take away from this. And I can't shake the feeling that most of the commenters have completely missed the point.
For me, the striking part of this story is that the Jester is shocked and confused when they drag him away. "How?!" He says "It's logically impossible". The Jester seems not to understand how it is possible for the dagger to be in the second box. My explanation goes as follows, and I think I'm just paraphrasing the king here.
1- If a king has two boxes and a means to write on them, then he can write any damn thing on them that he wants to. 2- If a king also has a dagger, then he can place that dagger inside one of the two boxes, and he can place it in whichever box he decides to place it in.
That's it. That's the entire explanation for how the dagger could "possibly" be inside the second box. It's a very simple argument, that a five year old could understand, and no amount of detailed consideration by a logician is going to stop this simple argument from being true.
The jester, however, thought it was impossible for the dagger to be in the second box. Not just that it wasn't there, but that it was IMPOSSIBLE. That's how I read the story, anyway. He used significantly more complicated logic, and he thought that he'd proven it impossible. But it only takes a moment's reflection to see that he's wrong.
Some of the comments above have tried to work out what was wrong with Jester's logic, and they've explained the detailed and subtle flaws in his reasoning. That's great - if you want to develop a deep understanding of logic, self-referential statements, and mathematical truth values (and lets be fair, I suppose most of us do), but in the context of the sequences on rationality, I think there's a much better lesson to learn.
Remember: rationalists are supposed to WIN. We're supposed to develop reasoning skills that give us a better and more useful understanding of reality. So the lesson is this: don't be seduced by complex and detailed logic, if that logic is taking you further and further away from an accurate description of reality. If something is already true, or already false, then no amount of reasoning will change it.
Reality is NOT required to conform to your understanding or your reasoning. It is your reasoning that should be required to conform to reality.
Breaking #24 of the Evil Overlord List makes me wince, too, even if it's a jester doing it. Not sure if that's the main point, though, but then, none of the proposed explanation for how the king could pull his "riddle" off without at any point lying feel entirely right to me, so, unless someone offers to help me, I shall have to take your advice and not let myself get entangled in the "complex and detailed logic", when the answer might as well be "BS".
Once upon a time, there was a court jester who dabbled in logic.
The jester presented the king with two boxes. Upon the first box was inscribed:
On the second box was inscribed:
And the jester said to the king: "One box contains an angry frog, the other box gold; and one, and only one, of the inscriptions is true."
The king opened the wrong box, and was savaged by an angry frog.
"You see," the jester said, "let us hypothesize that the first inscription is the true one. Then suppose the first box contains gold. Then the other box would have an angry frog, while the box with a true inscription would contain gold, which would make the second statement true as well. Now hypothesize that the first inscription is false, and that the first box contains gold. Then the second inscription would be—"
The king ordered the jester thrown in the dungeons.
A day later, the jester was brought before the king in chains, and shown two boxes.
"One box contains a key," said the king, "to unlock your chains; and if you find the key you are free. But the other box contains a dagger for your heart, if you fail."
And the first box was inscribed:
And the second box was inscribed:
The jester reasoned thusly: "Suppose the first inscription is true. Then the second inscription must also be true. Now suppose the first inscription is false. Then again the second inscription must be true. So the second box must contain the key, if the first inscription is true, and also if the first inscription is false. Therefore, the second box must logically contain the key."
The jester opened the second box, and found a dagger.
"How?!" cried the jester in horror, as he was dragged away. "It's logically impossible!"
"It is entirely possible," replied the king. "I merely wrote those inscriptions on two boxes, and then I put the dagger in the second one."
(Adapted from Raymond Smullyan.)