I think this line of thinking is very important. People would benefit immensely from becoming better at deciding what questions to address with their scarce cognitive resources. However, I do not think this problem of "meta-rationality" is an easy one, and in particular I'm not sure your heuristic is a good one. The principle that a good question has a clear-cut policy implication conflicts directly with the principle of curiosity. Maybe if an individual is an high-stress, high-stakes decision-making role, he or she may want to ignore questions that are not immediately relevant to the problems at hand. But the whole idea of academia is that society benefits when some individuals have the time and the incentive to go out and answer questions of academic interest - because, of course, we don't know what we don't know and some ideas, or their consequences, may have nonobvious policy implications somewhere down the road.
I propose the following heuristics, noting that in this area one should adopt a "fox-like" strategy and try to apply as many different perspectives as possible:
Does this question have a definitive answer? Will I know I have the right answer when I find it?
It is a good sign here if the answer is "yes". Mathematics as a field is worthwhile, in large part, because mathematicians know with a very high degree of confidence when they have produced a correct result (in contrast, say, to medical science).
Is this question in a reference class with other questions that led to important or significant answers?
For example, in the field of AI one of the most standard strategies is to try to take a key insight from some other domain of knowledge - economics, physics, evolution, etc - and try to apply it to the problem of intelligence (a famous immunologist, Gerald Edelman, has made significant efforts to apply his insights from immunology to the problem of consciousness; in computer vision there is a very well known paper about edge detection that is very clearly inspired by the path integral formulation of quantum mechanics). I personally believe that questions in this reference class don't typically yield much progress, but YMMV.
Is the mental algorithm I ran to conclude that this question is important vulnerable to "meta-bias", i.e., from meta-level analogues to known cognitive biases? For example, did I falsely conclude that this question is important because of group pressure and conformity effects?
I was hoping that starting with examples involving politics would make it clear that I wasn't suggesting we toss out intellectual curiosity, but I can make an edit clarifying this.
Related to: Privileging the Hypothesis
-- Paul Graham
-- Doug Henwood
-- Eliezer Yudkowsky
Here are some political questions that seem to commonly get discussed in US media: should gay marriage be legal? Should Congress pass stricter gun control laws? Should immigration policy be tightened or relaxed?
These are all examples of what I'll call privileged questions (if there's an existing term for this, let me know): questions that someone has unjustifiably brought to your attention in the same way that a privileged hypothesis unjustifiably gets brought to your attention. The questions above are probably not the most important questions we could be answering right now, even in politics (I'd guess that the economy is more important). Outside of politics, many LWers probably think "what can we do about existential risks?" is one of the most important questions to answer, or possibly "how do we optimize charity?"
Why has the media privileged these questions? I'd guess that the media is incentivized to ask whatever questions will get them the most views. That's a very different goal from asking the most important questions, and is one reason to stop paying attention to the media.
The problem with privileged questions is that you only have so much attention to spare. Attention paid to a question that has been privileged funges against attention you could be paying to better questions. Even worse, it may not feel from the inside like anything is wrong: you can apply all of the epistemic rationality in the world to answering a question like "should Congress pass stricter gun control laws?" and never once ask yourself where that question came from and whether there are better questions you could be answering instead.
I suspect this is a problem in academia too. Richard Hamming once gave a talk in which he related the following story:
Academics answer questions that have been privileged in various ways: perhaps the questions their advisor was interested in, or the questions they'll most easily be able to publish papers on. Neither of these are necessarily well-correlated with the most important questions.
So far I've found one tool that helps combat the worst privileged questions, which is to ask the following counter-question:
What do I plan on doing with an answer to this question?
With the worst privileged questions I frequently find that the answer is "nothing," sometimes with the follow-up answer "signaling?" That's a bad sign. (Edit: but "nothing" is different from "I'm just curious," say in the context of an interesting mathematical or scientific question that isn't motivated by a practical concern. Intellectual curiosity can be a useful heuristic.)
(I've also found the above counter-question generally useful for dealing with questions. For example, it's one way to notice when a question should be dissolved, and asked of someone else it's one way to help both of you clarify what they actually want to know.)