Quick note: I'm glad we're doing this. Let's keep going.

I guess at this point I understand my disagreement with Eliezer and even Paul relatively well but I don't know where the core of the disagreement between the two of us lies. Do you have a better idea of this?

Is it easy for you to sum up your core disagreements with Eliezer and with Paul? That would be pretty useful to my own strategic thinking.

As for where our core strategic disagreements are… I skimmed through your posts that looked fairly strategy-relevant: Cynical explanations of FAI critics, Work on security instead of Friendliness?, How can we ensure that a Friendly AI team will be sane enough, Reframing the problem of AI progress, Against "AI risk", Modest superintelligences, Wanted: backup plans for "see AI turns out to be easy", Do we want more publicity, and if so how?, Some thoughts on singularity strategies, Outline of possible singularity scenarios (that are not completely disastrous), Metaphilosophical Mysteries, Hacking the CEV for fun and profit, Late great filter is not bad news, Complexity of value ≠ complexity of outcome, Value uncertainty and the singleton scenario, Non-Malthusian scenarios, Outside view(s) and MIRI's FAI endgame, and Three approaches to "Friendliness".

My first idea is that we might disagree about the plausibility of the alternatives to "AI-foom disaster" you list here. My second idea is that some of our core disagreements are about the stuff we're talking about in this thread already.

I wonder why you don't post/crosspost those things here

I did this once and didn't get much of a response. But maybe I could do it more anyway.

No that's not clear. An almost-Friendly AI could be worse than a completely indifferent AI, since "bad" ...utility functions are [closer] in design space to "good" utility functions. I think somebody made this argument before on LW but I forgot who/where.

Right. I think I was persuaded of this point when we discussed it here. I think the question does deserve more analysis (than I've seen written down, anyway), but I could easily see it being one of the questions that is unwise to discuss in great detail in public. I'd definitely like to know what you, Eliezer, Carl, Bostrom, etc. think about the issue.

The key questions are: how much greater is P(eAI | FAI attempted) than P(eAI | FAI not-attempted), and what tradeoff are we willing to accept? The first part of this question is another example of a strategic question I expect work toward FAI to illuminate more effectively than any other kind of research I can think of to do.

I notice that many of your worries seem to stem from a worry not about the math work MIRI is doing now, but perhaps from a worry about mission lock-in (from cognitive dissonance, inertia, etc.). Is that right? Anyway, I don't think even Eliezer is so self-confident that he would make a serious attempt at Yudkowskian FAI even with mounting evidence that there was a pretty good chance we'd get eAI from the attempt.

There's also an argument that if acausal control/trading is possible

I have no comment on this part since I haven't taken much time to familiarize myself with acausal trade arguments. And I stand by that choice.

I think what may be useful is "AI safety" (for lack of a better term) research that is done explicitly under the assumption that we may have to deploy a not-quite-Friendly AI to head off a greater danger, which would involved approaches quite different Eliezer's current one

It's not MIRI's focus, but e.g. Carl has spent a lot of time on that kind of thing over the past couple years, and I'm quite happy for FHI to be working on that kind of thing to some degree.

In that case it seems like the cost of the strategic information you're seeking is really high

You might be underestimating how costly it is to purchase new strategic insights at this stage. I think Bostrom & Yudkowsky & company picked up most of the low-hanging fruit over the past 15 years (though, most of it hasn't been written up clearly anywhere). Example 1 (of difficulty of purchasing new strategic insights): Bostrom's book has taken many person-years of work to write, but I'm not sure it contains any new-to-insiders strategic insights, it's just work that makes it easier for a wider population to build on the work that's already been done and work toward producing new strategic insights. Example 2: Yudkowsky (2013)+Grace (2013) represents quite a lot of work, but again doesn't contain any new strategic insights; it merely sets up the problem and shows what kind of work would need to be done on a much larger scale to (maybe!) grab new strategic insights.

Also, remember that MIRI's focus on math work was chosen because it purchases lots of other benefits alongside some expected strategic progress – benefits which seem to be purchased less cheaply by doing "pure" strategic research.