LESSWRONG
LW

by [anonymous]
2 min read
12

29

Funding agencies like DARPA encourage you to consider a list of questions to guide the development of a good proposal. These are often called the Heilmeier questions or Heilmeier catechism. I often look back at this checklist when starting a new research project; they provide good inspiration, but as I read LW, I realize they could use reworking. These are the original questions:
  1. What are you trying to do? Articulate your objectives using absolutely no jargon.
  2. How is it done today, and what are the limits of current practice?
  3. What's new in your approach and why do you think it will be successful?
  4. Who cares?
  5. If you're successful, what difference will it make?
  6. What are the risks and the payoffs?
  7. How much will it cost?
  8. How long will it take?
  9. What are the midterm and final "exams" to check for success?
The best question is number 3, which I've seen in a more effective form as "Why now and not last year or ten years ago?" This is a perfect outside view question that forces you to confront the possibility that there are lots of smart people around, none of them has done this, and there is probably some good reason for this. You should know the reason before you start, and reasons like "They weren't smart enough" or "They simply never thought about this topic" are probably incorrect.
1,5,6,9 are OK but vague. "What would success look like? Is that what I want?" is a question often overlooked, even for projects that are personally very important, like picking a career. When I asked myself this question in grad school, I found myself pushed towards changing fields.
The worst are 7 and 8, which should be "How much have projects like this one cost? How long have projects like this one taken?"
The usefulness of such catechisms is a delicate balancing act. The questions should be broad enough to apply to many situations, but specific enough to start a specific train of thought for each input problem. 
A proposed, simplified revision. I would love feedback.
So you want to solve problem x using method y.
  1. What difference would it make if you solved this problem? Is that what you want?
  2. Why hasn't someone already solved this problem? What makes you think what stopped them won't stop you?
  3. Are there sub-problems to x? Repeat 1 and 2 for these sub-problems.
  4. Are there other potential methods to solve this problem? If so, why are you considering y and not the others?
  5. Are there other implications to solving x that you haven't considered? If so, go back to 1.
  6. If y fails to solve x, what would that teach you that you (hopefully) didn't know at the beginning?
Personal Blog

29

Mentioned in
The Heilmeier Questions
New Comment
12 comments, sorted by
top scoring
Click to highlight new comments since:
[-]jsalvatier130

Consider the question: "why this problem and not a different problem?", we should encourage people to optimize their problem selection more.

Reply
[-]SilasBarta20

Enforcing the asking of 1, 2, and 5 (in the first list) seems like it would eliminate 90% of what people are doing in academia.

Reply
[-]Daniel_Burfoot20

"If you invest N years of your life in this project and totally fail, will you be happy that you tried anyway?"

Reply
[-]Daniel_Burfoot130

It's part of an expected utility calculation. The other questions are relevant to the estimation of the probability of success (P) and the utility of success (A). That suggests the expected utility is PA. But actually the expected utility is PA + (1-P)B, where B is the utility of failure. The second term is probably non-trivial, and might even dominate. For further analysis of this issue, see here.

Reply
[-]JoshuaZ30

As a risk averse individual, I do.

Reply
[-]JGWeissman00

Your project should be funded on spec by investors/donors and essentially be part of a diversified portfolio held by a larger entitity than yourself as an individual. Your individual risk is far less because you are paid either way.

Reply
[-]rwallace00

But what if it's something you think is worth trying, yet investors are not convinced of this?

Reply
[-]JGWeissman00

If investors are not convinced (and you are not independently wealthy, in which case you should be less risk adverse), you will not be able to devote your full time efforts to a long term project, no matter how worthwhile you think it is.

Reply
[-]rwallace00

True, but there are gray areas in the middle ground: spend evenings and weekends on your research project, find part time employment that gives you a bit more spare time, do the research as a graduate student, live on the dole and eat noodles, ask family for support, take out a second mortgage on your house to pay the bills in the meantime, spend a year in Australia working as a bartender to save up enough money to cover a year of research, etc.

Of course none of these is ideal, and none is a permanent solution, but then, venture capitalists (let alone banks) usually won't finance research - they're looking for faster and surer payback, and even trying to do development as a startup is hard enough - so as a practical matter, most people with a research project they want to pursue end up having to try one or more of the gray area solutions at least for a while (unless you already have an established position in academia or a corporate think tank, and obtaining such a position is a hard and uncertain task with costs of its own).

So the truth is, unless you are particularly lucky, you're going to be paying a substantial personal cost one way or another, which doesn't mean you shouldn't do it, but it does mean is this worth the cost even considering that I might fail? is a highly relevant question.

Reply
[-]atucker10

How do you know if this solution works?

Reply
[-]Larifari00

I think something like question 9 is very helpful. Even more important than having an a priori success criterion, instead of defining it a posteriori to fit whatever one has accomplished (though in science sometimes redefining success criteria does make sense, because one does never know exactly in advance what can be accomplished), is the mental exercise of coming up with such a criterion. If one isn't even able to do this, that's a very bad sign for the project.

Reply
Moderation Log
12Comments