I find that all questions fall into one of 3 categories:
Well defined: These questions are clear, and contain all the basic information you need to answer them, with little or no need to infer what the questioner meant. Word problems are a good example, and so is someone asking for directions or asking what you would like for dinner.
Poorly defined: These are problems that you don't know how to solve, at least at first. Maybe you have to learn what the questioner means, or maybe you have to acquire some fundamental understanding in order to evaluate several possible methods of coming to a solution. Most engineering R&D problems fit in this box, but other examples include the chicken/egg problem and asking whether a tree falling in a forest makes a sound.
What is the square root of a potato: At least that's how I've always described such questions to people. These are questions that are so poorly defined that they are demonstrably completely incoherent. There is no possible way of interpreting these questions which could possibly have an answer. It's when a question is "not even wrong". This seems quite similar to what Yudkowsky is describing here, although maybe he is referring to things somewhere between #2 and #3. I guess you could create a spectrum running from #1 to #3 type questions, but I've found that problems tend to fall pretty neatly into these 3 categories, so I haven't found it useful to think of them as lying along a spectrum.
Where the mind cuts against reality's grain, it generates wrong questions—questions that cannot possibly be answered on their own terms, but only dissolved by understanding the cognitive algorithm that generates the perception of a question.
One good cue that you're dealing with a "wrong question" is when you cannot even imagine any concrete, specific state of how-the-world-is that would answer the question. When it doesn't even seem possible to answer the question.
Take the Standard Definitional Dispute, for example, about the tree falling in a deserted forest. Is there any way-the-world-could-be—any state of affairs—that corresponds to the word "sound" really meaning only acoustic vibrations, or really meaning only auditory experiences?
("Why, yes," says the one, "it is the state of affairs where 'sound' means acoustic vibrations." So Taboo the word 'means', and 'represents', and all similar synonyms, and describe again: How can the world be, what state of affairs, would make one side right, and the other side wrong?)
Or if that seems too easy, take free will: What concrete state of affairs, whether in deterministic physics, or in physics with a dice-rolling random component, could ever correspond to having free will?
And if that seems too easy, then ask "Why does anything exist at all?", and then tell me what a satisfactory answer to that question would even look like.
And no, I don't know the answer to that last one. But I can guess one thing, based on my previous experience with unanswerable questions. The answer will not consist of some grand triumphant First Cause. The question will go away as a result of some insight into how my mental algorithms run skew to reality, after which I will understand how the question itself was wrong from the beginning—how the question itself assumed the fallacy, contained the skew.
Mystery exists in the mind, not in reality. If I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon itself. All the more so, if it seems like no possible answer can exist: Confusion exists in the map, not in the territory. Unanswerable questions do not mark places where magic enters the universe. They mark places where your mind runs skew to reality.
Such questions must be dissolved. Bad things happen when you try to answer them. It inevitably generates the worst sort of Mysterious Answer to a Mysterious Question: The one where you come up with seemingly strong arguments for your Mysterious Answer, but the "answer" doesn't let you make any new predictions even in retrospect, and the phenomenon still possesses the same sacred inexplicability that it had at the start.
I could guess, for example, that the answer to the puzzle of the First Cause is that nothing does exist—that the whole concept of "existence" is bogus. But if you sincerely believed that, would you be any less confused? Me neither.
But the wonderful thing about unanswerable questions is that they are always solvable, at least in my experience. What went through Queen Elizabeth I's mind, first thing in the morning, as she woke up on her fortieth birthday? As I can easily imagine answers to this question, I can readily see that I may never be able to actually answer it, the true information having been lost in time.
On the other hand, "Why does anything exist at all?" seems so absolutely impossible that I can infer that I am just confused, one way or another, and the truth probably isn't all that complicated in an absolute sense, and once the confusion goes away I'll be able to see it.
This may seem counterintuitive if you've never solved an unanswerable question, but I assure you that it is how these things work.
Coming tomorrow: A simple trick for handling "wrong questions".