One possible answer to the argument "attempting to build FAI based on Eliezer's ideas seems infeasible and increases the risk of UFAI without helping much to increase the probability of a good outcome, and therefore we should try to achieve a positive Singularity by other means" is that it's too early to decide this. Even if our best current estimate is that trying to build such an FAI increases risk, there is still a reasonable chance that this estimate will turn out to be wrong after further investigation. Therefore, the counter-argument goes, we ought to mount a serious investigation into the feasibility and safety of Eliezer's design (as well as other possible FAI approaches), before deciding to either move forward or give up.
(I've been given to understand that this is a standard belief within SI, except possibly for Eliezer, which makes me wonder why nobody gave this counter-argument in response to my post linked above. ETA: Carl Shulman did subsequently give me a version of this argument here.)
This answer makes sense to me, except for the concern that even seriously investigating the feasibility of FAI is risky, if the team doing so isn't fully rational. For example they may be overconfident about their abilities and thereby overestimate the feasibility and safety, or commit sunken cost fallacy once they have developed lots of FAI-relevant theory in the attempt to study feasibility, or become too attached to their status and identity as FAI researchers, or some team members may disagree with a consensus of "give up" and leave to form their own AGI teams and take the dangerous knowledge developed with them.
So the question comes down to, how rational is such an FAI feasibility team likely to be, and is that enough for the benefits to exceed the costs? I don't have a lot of good ideas about how to answer this, but the question seems really important to bring up. I'm hoping this post this will trigger SI people to tell us their thoughts, and maybe other LWers have ideas they can share.
My guess was that you think the things on this list are more probable and/or bad than I do, especially the first three. But to keep things focused, I suggest we not add those possible disagreements to the current thread.
In part, I'm also trying to see whether we can cause discussion to happen on MIRI's blog. If not, we'll have to do it on LW. But that'll be better when we've had enough development resources go into LW that we can have subreddits (e.g. one for astronomical stakes stuff). Right now the thing under development is a dashboard that will make it easier to follow the content you want to follow across multiple subreddits, and then after that we plan to create subreddits so that people can more easily follow only the things they care about, rather than having everything get dumped into Discussion, such that busier and less interested people can't be bothered to follow everything in order to occasionally find snippets of things they care about.
Ok, gotcha. Since you're in town in a few days, Eliezer suggested that we three chat about this when you get here. See you soon!
My impression is that the ideas in your strategic posts are mostly ones already considered by Eliezer, Carl, Bostrom, and perhaps other people, though the three of them could confirm or deny this. And there's a lot more that hasn't been written down. E.g. whenever we have strategy meetings at MIRI, we very quickly find ourselves discussing arguments that people like Eliezer and Carl have known about for 5+ years but which haven't been written down anywhere that I know of.
Why haven't these things been written down? Because there's so much to write up, because other projects often seem like higher value, because sometimes people are hesitant to publish half-baked reasoning under their own name… lots of reasons. I've been passionate about getting things written up since I arrived in April 2011, and tried lots of different things to get the people who know this stuff to write it down, but it's been a very slow process with many detours, and in the end I've had to do quite a bit of it myself. Beckstead's thesis helps. Bostrom's book will help, too.
If any of you three have the time to write up a brief summary of that chat afterwards, I expect that a lot of people would be interested in reading that. (Where "a lot of people" = "I know that I would be, and generalizing from my own example obviously means that other people must be, too". :-))