Stupid Questions Open Thread Round 4

lukeprog

This is for anyone in the LessWrong community who has made at least some effort to read the sequences and follow along, but is still confused on some point, and is perhaps feeling a bit embarrassed. Here, newbies and not-so-newbies are free to ask very basic but still relevant questions with the understanding that the answers are probably somewhere in the sequences. Similarly, LessWrong tends to presume a rather high threshold for understanding science and technology. Relevant questions in those areas are welcome as well. Anyone who chooses to respond should respectfully guide the questioner to a helpful resource, and questioners should be appropriately grateful. Good faith should be presumed on both sides, unless and until it is shown to be absent. If a questioner is not sure whether a question is relevant, ask it, and also ask if it's relevant.

Ask away!

Previously: round 1, round 2, round 3

From the original thread:

This is for anyone in the LessWrong community who has made at least some effort to read the sequences and follow along, but is still confused on some point, and is perhaps feeling a bit embarrassed. Here, newbies and not-so-newbies are free to ask very basic but still relevant questions with the understanding that the answers are probably somewhere in the sequences. Similarly, LessWrong tends to presume a rather high threshold for understanding science and technology. Relevant questions in those areas are welcome as well. Anyone who chooses to respond should respectfully guide the questioner to a helpful resource, and questioners should be appropriately grateful. Good faith should be presumed on both sides, unless and until it is shown to be absent. If a questioner is not sure whether a question is relevant, ask it, and also ask if it's relevant.

Ask away!

Solving this problem alone could easily take a team several years to accomplish, so how do you hope to produce the strategic recommendations, which must take into account many such issues, in 2 years?

Two answers:

Obviously, our recommendations won't be final, and we'll try to avoid being overconfident — especially where the recommendations depend on highly uncertain variables.
In many (most?) cases, I suspect our recommendations will be for policies that play a dual role of (1) making progress in directions that look promising from where we stand now, and also (2) purchasing highly valuable information, like how feasible an NGO FAI team is, how hard FAI really is, what the failure modes look like, how plausible alternative approaches are, etc.

SI, FHI, you, others — we're working on tough problems with many unknown and uncertain strategic variables. Those challenges are not unique to AI risk. Humans have many tools for doing the best they can while running on spaghetti code and facing decision problems under uncertainty, and we're gaining new tools all the time.

I don't mean to minimize your concerns, though. Right now I expect to fail. I expect us all to get paperclipped (or turned off), though I'll be happy to update in favor of positive outcomes if (1) research shows the problem isn't as hard as I now think, (2) financial support for x-risk reduction increases, (3) etc.

Right now I expect to fail. I expect us all to get paperclipped

So, you're currently thinking hard about the best way to approach someone like Terence Tao? (Doesn't have to be him, someone else's blog might also have comments and give you a better opportunity to raise the issue.)

2Wei Dai14y

I think you may have misunderstood my intent here. I'm not trying to make you more pessimistic about our overall prospects but arguing (i.e., trying to figure out) the absolute and relative importance of solving various strategic problems. Another point was to suggest that perhaps SI ought to give higher priority to recruiting/training "hero strategists" as opposed to "hero mathematicians". For example your So You Want to Save the World says: [...] which fails to credit the importance of strategic contributions (even though later in the post there is a large section on strategic problems).

9

Stupid Questions Open Thread Round 4

9

9

9

Stupid Questions Open Thread Round 4

9

9