I recently got a chance to interview a couple people about this who'd done product management or similar at bay area tech companies.

They agreed that you can't run projects there unless you project near-certainty the project will succeed. However, they had a trick that had failed to occur to me prior to them saying it, which is to find a mid-scale objective that is all of: a) quite likely to have at least a bit of use in its own right; b) almost certainly do-able; and c) a stepping-stone for getting closer to the (more worthwhile but higher-failure-odds) goal. And work towards your harder goal via stepping-stones such as these, when doing product management for groups.

I'll be trying this out, probably. It reminds me of building modular code instead of spaghetti code.

Less confidently, I'd like to lightly recommend reading this blog post about Christopher Alexander's notions of how to make buildings into places where humans like to be, and about analogous ideas for how to do good software design. I don't have concrete take-aways to point to from that one, so read it at your own risk or don't, but it seems to me there may be patterns like this to how to make goals and subgoals that groups can have morale/orientation while building.

I'm a bit surprised that most of the previous discussion here was focused on the "okay, so how do you actually motivate people?" aspect of this post.

This post gave me a fairly strong "sit bolt upright in alarm" experience because of it's implications on epistemics, and I think those implications are sneaky and far reaching. I expect this phenomenon to influence people's ability to think and communicate, before you get to the point where you actually have a project that people are hitting the hard parts of. 

People form models of what sort of things they can achieve, and what sort of projects to start, and how likely their friends are to succeed at things, and (I expect) this backpropagates through their entire information system.

The problem may not just affect the people on a given project – it could affect the people nearby (and the people nearby them in turn), in terms of what sort of feedback you can easily give each other. 

I've struggled with deciding whether to give people feedback that I don't think their project is likely to work, when I nonetheless think the project is the right thing to be working on, either for EV reasons or longterm-growth-as-a-person reasons, i.e. their next project will benefit from the skills they're gaining here. I know there are some people who'd lean hard into the "information is a key bottleneck, definitely don't withhold info like that." And they may be right. But if so, that still leaves us with a deep, crucial question of how to integrate epistemics and actual success on hard, long problems that lets you actually win at the instrumental part.

(I don't think the "pay people a lot" is actually sufficient, although it obviously helps. I think lack-of-clear-path-to-victory can be demoralizing even if you have plenty of money – that seems to be where a lot of 20th century ennui comes from. Also, many of the key projects here are early-stage where it's just hard to convince people to give you enough money to escape scarcity mindset)

I notice I don't have a very explicit model of what's going on – this comment felt more like I was ranting than clearly explaining things, not sure how it comes across to others. 

I hope to flesh this out (and hopefully make some empirical predictions) during the review phase.

Hmm, I worry that motivation is only part of the picture. There's also idiosyncrasy between agents in terms of ability and acceptance of outcome.

By this I mean that they would not have stuck to writing the Sequences or HPMOR or working on AGI alignment past the first few months of real difficulty,

True enough, though you can replace "would not" with "will not" or "did not".

without assigning odds in the vicinity of 10x what I started out assigning that the project would work

This is not the counterfactual I'd assign the most weight to. "Without being Eliezer" is probably too specific, but "without having Eleizer's history of rewarded iconoclasm" wouldn't be a stretch. It's extremely likely that you _ARE_ orders of magnitude more likely to succeed at these endeavors than the majority of people who say they're interested in the topic.

I'm painfully familiar with the issue of lack of group participation, since I can't even get people to show up to a meetup.

Because of that, I've been doing research on identifying the factors contributing towards this issue and how to possibly mitigate them. I'm not sure if any of this will be new to you, but it might spark more discussion.
These are the first ideas that come to mind:

1. For people to be intrinsically motivated to do something, the process of working on it has to be fun or fulfilling.

2. Extrinsic motivation, as you say, requires either money or a reason to believe the effort will accomplish more than other uses of one's time would. If it's a long-term project, the problem of hyperbolic discounting may lead people to watch TV or [insert procrastination technique here] instead, even if they think the project is likely to succeed.

3. If people already have a habit of performing an activity, then it takes less effort for them to participate in similar activities and they demand less benefit from doing so. Identifying habits that are useful for the task you have in mind can be tricky if it's a complex issue, but successfully doing so can keep productivity consistent and reduce its mental cost.

4. Building a habit requires either intense and consistent motivation, or very small steps that build confidence. Again, though, identifying very small steps that still make for good productivity early on may be tricky.

5. If you have trouble getting people to start joining, it may be good to seek out early adopters to provide social proof. However, the social proof may only work for people who are familiar with those specific early adopters and who take cues from them. In that case, you may need to find some regular early adopters and then identify trendsetters in society (habitual early adopters from whom many people take their cues) you could get on board, after which their followers will consider participating. (Then the danger becomes making sure that the participants understand the project, but at least you have more people to choose from.)

6. It may help to remind people from time to time what they're working towards, even though everyone already knows. Being able to celebrate successes and take some time to review the vision can go quite a ways in relieving stress when people start to feel like their work isn't rewarding.

From item 1, if people think they can get a benefit from working on a project even if the project fails, they might be willing to participate. Socializing with project members and forming personal relationships with them may help in this respect, since they'll enjoy working with people. Alternatively, you could emphasize the skills they'll pick up along the way.

From item 4, I've been working on 'mini-habits" (a concept I got from Stephen Guise) to lower my own mental costs for doing things, and it seems to be working fairly well. Then the trick becomes getting enough buy-in (per item 5) so you can get other people started on those mini-habits.

There are probably some other factors I'm overlooking at the moment. Since I haven't been able to get results yet, I can't say for sure what will work, but I hope this provides a helpful starting point for framing the problem.

This post has influenced my evaluations of what I am doing in practice by forcing me to consider lowering the bar for expected success for high return activities. Despite "knowing" about how to shut up and multiply, and needing to expect a high failure rate if taking reasonable levels of risk, I didn't consciously place enough weight on these. This helped move me more in that direction, which has led to both an increased number of failures to get what I hoped, and a number of mostly unexpected successes when applying for / requesting / attempting things.

It is worth noting that I still need to work on the reaction I have to failing at these low cost, high-risk activities. I sometimes have a significant emotional reaction to failing, which is especially problematic because the emotional reaction to failing at a long-shot can influence my mood for multiple days or weeks afterwards.

I love how cleanly this brings up its point and asks the question. My answer is essentially that you can do this if and only if you can create expectation of Successful Failure in some way. Thus, if failing person's real mission can be the friends they made along the way or skills they developed or lessons learned, or they still got a healthy paycheck, or the attempt brings them honor, or whatever, that's huge.

Writing a full response is on my list of things to eventually do, which is rare for posts that are over a year old.

Focusing on the chance of success may be missing other factors. You need to also consider their belief in the chance that the problem is real and their belief that someone else won't solve it before them.

Someone who has, say a 30% belief in their ability to contribute to solving a problem will have very different motivations if they're 90% vs 10% confidant that the problem is real (and important) and if they have a 90/10% belief that someone else is going to solve it first.

I doubt it's as simple as multiplying those estimates, but that's an ok place to start.

From my experience with working with groups I've found a few things to work, all of them revolve around incentivizing or promoting positive feedback loops. This includes: bribery/payment (while most people think of this as being with money, since I work with 10-17 year olds it's usually candy or pizza), lowering the bar so that small bits of progress are celebrated greatly, making the actual task less boring (turning on some music, working with friendly people, making it look pretty), humor, taking breaks to promote sanity, having a comfortable working environment etc.

The core idea of this post has occurred to me more than once when considering plans. I'm still working on how to relate to plans with low chances of success. On the one hand low chances of success suggest a bad plan, and being willing to "do the improbable" feels like it's an excuse for having a bad plan. On the other, sometimes you really want to be doing low probability, high EV plans. I'm uncertain over whether LW counts as this. Sometimes I think we can definitely succeed at big stuff, sometimes it more seems like high-EV low probability. I'm not sure.

But all in all, the idea here seems important to think more about. I'd actually like to see more thought on this (and perhaps do my own stuff here).

Also want to note that Anna's post on "going at half-speed" seems at least a bit related.